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Executive summary 


The project 


Challenge the Gap (CtG) is a school collaboration programme designed by Challenge Partners that 
aims to break the link between disadvantage and attainment. The main components of CtG are: 


e after-school workshops drawing on published research and evidenced practice; 
e focused in-school interventions with a selected cohort of disadvantaged pupils; 
e cross-school collaboration and practice development; and 

e practical tools and resources for use in schools and additional online materials. 


Each school is assigned to a ‘trio’, comprising one ‘Lead’ school, which has demonstrated effective 
practices for reducing the attainment gap, and two ‘Accelerator’ schools, for which closing the 
attainment gap is a major priority. The programme lasts for a year, during which expertise is transferred 
from Lead to Accelerator schools. Schools initially target programme activity at a small group of 
disadvantaged students (12-15 in each school), with a view to rolling out the best practice more widely 
across the school and sharing what they have learned with new partner schools in subsequent years. 
The different approaches in each Lead school mean that practices vary between each trio. 


This project evaluated the 2012 version of the CtG programme (Challenge Partners have since 
developed and continue to develop the approach). An initial pilot study began in late 2012 and focused 
on the feasibility of the approach with a group of 30 schools. The next phase of the evaluation involved 
104 primary and secondary schools (21,041 pupils), which were funded by the Education Endowment 
Foundation (EEF) to take part in the programme between 2013 and 2014, and then given the option to 
pay for a further year between 2014 and 2015, taken up by 49 schools. The evaluation assessed the 
impact on all participating schools, using 2015 Key Stage 2 or Key Stage 4 results. The main evaluation 
was a matched-controlled efficacy trial. A randomised design was not feasible as the school-to-school 
nature of the work meant that schools would need to be recruited and randomised as groups of three. 
Process evaluation data was collected through surveys, school visits, interviews, and monitoring 
postings on the CtG website. 


Key conclusions 


1. The project found no evidence that Challenge the Gap (CtG) increased average attainment for 
either primary or secondary school pupils, overall. The security of the primary school results is low 
to moderate, and the security for the secondary school results is low. 


2. The findings are different for children eligible for free school meals. FSM-eligible children in CtG 
primary schools made 2 months’ additional progress compared to similar children in other schools, 
while FSM-eligible children in CtG secondary schools made 2 months’ fewer progress compared 


to similar children in other schools. The smaller number of FSM-eligible students in the trial means 
that these results are less secure than the overall findings. 


3. Teachers and non-teaching staff from participating schools were extremely positive about the 
involvement of their schools in the CtG programme and valued both the professional development 
opportunities it provided and the opportunity to collaborate with partner schools. 


4. CtG is a flexible programme that allows lead schools to share best practice. There was large 
variation between the strategies that were adopted by the schools in the programme. 


EEF security rating Primary school: & @ Secondary school: & 


The security rating of the trial indicates how confident we can be that any additional progress 
experienced by the children receiving CtG was due to CtG and not to any other factors. This trial was 
an efficacy trial, which tested whether the intervention can work under ideal or developer-lead 
conditions. The primary school findings have low to moderate security, while the secondary school 
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findings have low security. The trial was a large but weakly matched comparison design. Fewer than 
5% of pupils were not included in the analysis for reasons such as moving to another school. In the 
primary schools that took part in the trial, the pupils who received the intervention were similar to the 
pupils in the comparison group. In the secondary schools there were some important differences in 
GCSE grades between the schools that received the intervention and schools in the comparison group, 
which resulted in the lower security rating. 


Additional findings 


CtG is designed to narrow the attainment gap. Within primary schools the evaluation detected two 
months’ additional progress on the Key Stage 2 scores of children eligible for free school meals. This 
is in contrast with the analysis of the whole cohort, which showed no additional impact. This suggests 
that there has been a small reduction in the attainment gap and is consistent across lead and 
accelerator schools. The overall primary school findings, however, are of low to moderate security, 
meaning that we have limited confidence that they happened as a result of CtG. FSM findings involve 
fewer pupils and are likely to be less secure than the overall finding. In secondary schools there was 
no impact on the cohort overall, and a negative impact (also of two months) on the progress of children 
eligible for free school meals. This indicates that the gap might be widening in secondary schools. The 
secondary school results are of low security, meaning that we cannot be confident that they happened 
as a result of the intervention. 


Teachers and non-teaching staff from participating schools were extremely positive about CtG. 
Perceived benefits included links to the professional development and leadership development 
opportunities CtG provided, and benefits gained from collaboration with partner schools. These factors 
were outside the scope of the quantitative evaluation. Teachers also felt that participation in CtG led to 
an improvement of variables associated with academic performance, such as attitudes to school, and 
attendance. 


The first year of the intervention was funded by the EEF. After this, schools had the option of self- 
funding to remain in the programme for a subsequent year. Of the 104 schools that took part initially, 
49 chose to pay for the intervention over the second year. The main analysis was conducted on all 
schools after two years. Separate analysis for the 49 paying schools did not identify any additional 
benefit. 


Cost 


The annual subscription fee for Challenge the Gap is currently £4,975 for primary schools, and £8,500 
for secondary schools. Costs can be reduced for schools that meet criteria to host workshops for other 
schools. Assuming that all pupils in a school ‘benefit’ from CtG, and using the current annual 
subscription fees, the average cost per pupil per year is £19 in primary schools and £9 in secondary 
schools. 


Table 1: Summary of main outcomes 


Estimated EEF EEF 
Effect size months’ _ security dh hedia Pvalue cost 
progress e-lilaye} rating 
All pupils 0.00 
aa 
(Key Stage 2) (-0.04 to 0.03) 7 0.993 | £ 
FSM pupils 0.10 
(Key Stage 2) (0.05 to 0.15) na | 0.025 | £ 
All il 0.01 Efficacy 
pupils Et 
(Key Stage 4) (-0.03 to 0.02) 0 - 0.926 £ 
FSM pupils -0.10 
(Key Stage 4) (-0.14 to -0.05) 2 na 0.006 | £ 
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Introduction 


Intervention 


Challenge the Gap is a two-year school improvement programme that aims to help schools break the 
link between disadvantage and attainment. Developed from experiences of school-to-school 
collaboration within the London Challenge, the programme encourages schools to work together, 
sharing practice and engaging in joint practice development in order to accelerate attainment among 
disadvantaged pupil groups. During the period when the qualitative evaluation was conducted, 
participating schools were classified as either Lead or Accelerator schools. Lead schools, as indicated 
above, were schools that had demonstrated effective strategies for reducing the attainment gap; 
Accelerator schools were schools where closing the gap remained a priority. 


Schools in the pilot phase did not pay a fee. During the extension phase, Accelerator schools that 
recruited two additional schools (and became a Lead school) were eligible for a fee reduction in 
recognition of their contribution to the development of the new Accelerator schools. From 2015/16 the 
notion of Lead schools was discontinued, and the current yearly subscriptions for participation in the 
programme are £4,975 for primary schools and £8,500 for secondary schools. 


During the pilot phase, the programme had four main components: 


e workshops, delivered largely by experienced practitioners that draw on findings from 
published research and studies highlighting evidenced practice; 

e focused in-school intervention activities with a selected cohort of disadvantaged pupils; 

e cross-school collaboration and practice development within a trio of schools; and 

e visits to participating schools by CtG facilitators. 


Schools were expected to work together in trios. Each trio contained a Lead school, which had a strong 
record of progress for disadvantaged groups, and two Accelerator schools, that had identified closing 
the attainment gap as a major priority. The programme, which initially focuses on a target cohort of 
approximately 12-15 pupils in each school, also enables participating trios of schools to pursue 
improvements through wider collaboration within a local cluster — a wider group of schools that work 
together on the same goals and offer access to each other’s practices and support. 


In the extension phase, the cascading of the model was based on local clusters of schools, again 
working in trios led by a school that had established expertise in raising attainment among pupil 
premium children, but also ideally schools that already had experience of participation in the CtG 
programme as Accelerator schools. Within each cluster, there were between three and five trios. As in 
the pilot year, trio schools worked together offering school-to-school support and joint development 
opportunities to staff and, in some cases, pupils too. The Lead schools — initially these were schools 
with experience of working collaboratively to raise attainment, for example in the London Challenge, 
though subsequently they were recruited as outstanding local schools — had all demonstrated that they 
had appropriate strategies for raising attainment among all children from low income families. While 
during their first year of participation all trio schools focused efforts on a target cohort of approximately 
15 pupils (criteria for identifying particular pupils varied between schools) thus ‘enabling a highly 
targeted approach to improving attainment (CtG 2015), ultimately the aim was to raise the attainment 
of disadvantaged pupils more widely, and indeed to use knowledge gained to improve attainment of all 
pupils in the school. The small cohort size was intended to make ‘the programme more manageable 
and enable the team to understand the pupils as individuals, gaining clear insights into their issues and 
needs’ (ibid). Once successful strategies and approaches had been developed, these could be rolled 
out more widely, both across the school and, potentially, across the trio. In the second year of 
participation, it was expected that successful practices would be disseminated more widely as they 
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were cascaded into new trios, formed around participating schools that moved from being Accelerators 
to become Lead schools. 


Groups of three are a recurring feature in the way CtG seeks to organise participants. Each participating 
school identifies a ‘workshop team’, consisting of a member of the school leadership team and 
representatives from teaching staff and non-teaching staff groups in the school (one of each from 
primaries, two/three from secondary schools), who attend the workshop programme. The pilot 
workshop programme began with an induction meeting for the school leaders, which was followed by a 
series of workshops (six days in total) for each of the three staff groups (leaders, teachers, and 
paraprofessionals), though these were conducted as separate sessions for each of the three staff 
groups from participating schools. During workshops the CtG facilitators and guest speakers presented 
key research findings. There were also opportunities for school-level and cross-school planning, and to 
evaluate progress. During this pilot year the separate, whole-day meetings that were arranged for 
school leaders, teachers, and para-professionals, posed considerable organisational difficulties, and as 
the programme developed, the training workshops were rationalised into six core days attended by all 
staff, though during some workshops groups are split and separate sessions offered to the different 
staff groups. Currently the programme involves six sessions for primary schools and six for secondary 
schools spread out over the school year. The workshops follow a similar pattern to the pilot year, with 
the first taking place before the end of the school year so that both staff and target pupils can make a 
start during the summer break. Focus and content of sessions are: 


e Workshop 1 — The Starting Line 


. Introduction to the programme — moral purpose: National data vs. school 
data; trio working; audit/ evaluate current practice; target groups and summer 
activities 

e Workshop 2 — Focus on success 
. What does our data tell us about target pupils; what does the research say; 


identifying what needs to be done; introducing programme tools 
e Workshop 3 — The academically competent learner 
° Academic literacy: Building academic literacy in the classroom; 
interventions and engaging parents; comparing action plans, sharing through 
trio-to-trio carousels. Skills development for para-professionals (e.g. teaching 


assistants) 
e Workshop 4 — The self-aware learner 
° The pro-active learner: Metacognition and improving outcomes; 


working together — joint practice development; carousels; skills development 
para-professionals 
e Workshop 5 — The resilient learner 
. Resilience: Understanding pupil progress and overcoming barriers; 
building resilience in pupils; leading and managing change; carousels 
e Workshop 6 — So what and what next? 
. Learning from the previous year; preparing to extend reach in school in Year 
2; preparing to work with new teams 


During 2015/16 primary school participants attended only four workshops: workshops one, two and six, 
and then selected one of workshops three, four or five, depending on their main focus for the year, but 
now all schools follow a similar pattern. Schools also receive one visit during the year from a CtG 
facilitator. 


The workshops present current research findings, and encourage each school team to develop 
evidence-based strategies that can accelerate the progress of their selected target cohort. In doing this, 
teams are supported by collaboration with trio schools and CtG’s ‘expert practitioners’, who manage 
the workshop training sessions. Practical tools and resources are made available for use in the schools, 


Education Endowment Foundation 4 


Challenge the Gap 


and these are further supported by materials that are available online, via the CtG website that was 
established during the pilot phase and has since been populated by materials and examples both from 
CtG and from participating schools. Each workshop team member is paired with two colleagues from 
the same staff group back in school, and there is a clear expectation that the learning taken away from 
the workshop is spread within these in-school staff trios. 


This approach, combining knowledge and skills development through the workshop programme with 
what are essentially action research activities conducted back in school with the target cohort, provides 
a vehicle for research-based professional development, which is further enhanced by the cross-school 
collaboration within the trio of schools that encourages the spread of learning and effective practice 
between schools. These activities also seek to build the capacity of the staff members involved to lead 
change efforts within their own schools. This capacity then facilitates the rolling out of the programme 
more widely across pupil and year groups in the second year of the participation, when the practices 
developed while working intensively with a small target cohort of pupils in the previous year are spread 
out. 


Facilitation schools play a crucial role in the continued development of the programme. Many are 
teaching schools, and all are members of the Challenge Partners school support network, which 
initiated the CtG project. In the pilot phase, the majority of schools taking part were secondary schools, 
with only a small number of primary schools, most often integrated into cross-phase school trios. But 
by 2014/15 there were three primary facilitator schools, each leading primary school clusters. There 
were still some mixed trios in clusters led by secondary facilitator schools, but in the clusters led by 
primary schools all trios were made up of primary schools. 


The programme design assumes that participants will engage in activities at three levels: 1) directly 
within their own school, supporting the accelerated progress of target pupils and building a team to 
carry forward and extend this work across pupils and year groups; 2) inter-school activities within the 
trio schools, to enable practice to develop and spread across schools; and 3) workshop activities, which 
can become a source of ideas about and a catalyst for change and improvement through the 
opportunities for wider trio-to-trio exchange. The need for regular monitoring and reflection on what is 
being learned is a particular feature of the workshops, which ask for regular reports on progress and 
actions in school. Within participating schools the experiences and progress of the selected pupil cohort 
are closely monitored from the start of the programme. Most often this begins with an analysis of 
learning needs and the setting of challenging targets, pursued through appropriate interventions and 
support, with impact and progress carefully monitored. The expectation is that as ‘refined and new 
insights are gained, the programme is rolled out more widely to benefit many more pupils — pupil 
premium and non-pupil premium’ (CtG, 2015). 


This underlines the initial expectation that participating schools would remain in the programme for a 
second year, during which they would each themselves recruit two new schools to form a new trio thus 
expanding the programme. However, while this did happen in many cases, there were also many where 
it did not, either because schools were unable to recruit new partners or for other reasons chose not to 
continue. 


Background evidence 


The roots of this programme are firmly located within the London Challenge, though the methods and 
approaches used there were broadly repeated also in the Greater Manchester Challenge and the Black 
Country Challenge. As Evaluation of the City Challenge Programme (Hutchings et al., 2012) reported, 
working with partner schools was identified by teachers as a key factor for successful intervention: 


“... working with other headteachers and other school staff was rated as the most effective form of support for 
closing attainment gaps. In most cases, the cluster working that took place appeared to involve discussing and 
sharing practice rather than collaboration. The most effective cluster groups raised awareness of attainment gaps 
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between disadvantaged pupils and their peers ... (and) through discussion, enabled participants to come up with 
more imaginative and innovative ways of addressing gaps than might otherwise have been the case....” 
(Hutchings et al., 2012, p.80) 


It is clear that the cluster groups in CtG re-create this opportunity for working together (though it is less 
clear what the difference is between ‘sharing practice’ and ‘collaboration’). However, more generally 
the programme strategy reflects those aspects of the Challenge programmes judged by teachers to be 
most helpful, that is: clear structures and leadership, making sure all staff members are aware of the 
attainment gaps relating to disadvantage, working with other schools, drawing on materials developed 
in other projects to tackle the problem, but having the confidence to develop school-specific materials 
too (Hutchings et al., 2012). All of these issues are covered in the training workshops and are clearly 
signposted in the programme approach. 


At the same time, while it is evident from the above description that the programme grew out of specific 
strategies to accelerate pupil attainment nurtured largely within the City Challenges, the CtG 
programme is also firmly located within established school improvement practices and approaches, 
rather than a specific common ‘treatment delivered in the same fashion under tightly controlled 
conditions in a variety of locations. This means that we can expect to see differences in the 
implementation model at the local level, differences that may be critical to the success or failure of the 
programme and which make it difficult to tease out the relationship between means and ends. What is 
clear is that, despite the differences in local contexts and implementation approaches, there are some 
common elements that feature in each participating school’s own implementation strategy. In seeking 
to assess the impact of the programme on participating schools, we have chosen to focus on three such 
features, all of which are aspects of school improvement programmes that have been identified as key 
determinants of impact, and have been widely discussed in the literature. These are the nature of 
intervention itself, the role collaborative inquiry plays within implementation within the school, and the 
value added by school-to-school cooperation and networking. We will consider each of these in turn. 


Interventions to improve schools are by no means new, though the scope has widened and the pace 
quickened alongside the preoccupation with the measured outcomes of schooling. As we have 
previously noted (Kerr and West, 2010), while there are a number of ways interventions to improve 
schools might be conceptualised, Challenge the Gap is essentially a within-school intervention, that is, 
an approach driven by the evidence that pupil outcomes show significant within-school variation. 
Reviewing the evidence regarding interventions to improve schools, it is clear that some have more 
impact than others. Looking at the factors associated with such interventions being judged effective, we 
have noted, inter alia, that: 


e Interventions to improve schools need to be planned within the local context in which schools 
find themselves: one size does not fit all. 

e School leaders and teachers, who are closest to and in a better position to understand local 
contexts, are best placed to be able to shape and direct the intervention. 

e School improvement interventions that both take account of research evidence regarding 
‘what works’ and also use strategies that actively engage staff in planning and implementing 
any changes, are more likely to secure sustained improvement. 

e There is some evidence that authentic school-to-school collaboration can both promote and 
sustain developments in practice, though less is known regarding how authentic collaboration 
between schools is developed. 

e It may take some time before impact is reflected in pupil outcomes, so it is important to 
maintain focus, rather than leap from one intervention to another. 


(Kerr and West, 2010) 


The role played by collaborative inquiry by teachers in school improvement processes has also received 
increasing attention in recent years, and we have documented elsewhere why this seems to be a 
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sensible and practical arrangement for school improvement (Ainscow et al., 2012). Chapman et al. 
(2014) argue that adopting a collaborative inquiry approach to improvement is an effective strategy for 
tackling the attainment gap because: 


e it generates a commitment to long-term sustainability of performance improvement, extending 
beyond the time-frame of a particular project; 

e itis an approach underpinned by the creation of leadership and professional learning 
opportunities for staff; 

e it can deepen trust and relationships and build confidence to take risks and innovate; 

e it stimulates the use of systematic, focused practitioner enquiry to develop innovative 
practices; and 

e it encourages those involved to monitor the impact of their development. 


Such an approach requires new forms of relationships between practitioners and researchers, as 
outlined by Hiebert et al. (2002). They suggest that fruitful forms of collaboration require a re-orientation 
of values and goals among participants, with teachers moving away from the view that teaching is a 
‘personal and private activity’. 


The first benefit of collaborative school inquiry groups is that — for the members at least — it becomes 
possible to identify clear areas of focus for these efforts, so that they work systematically to achieve 
defined goals, rather than spreading their efforts thinly over too many activities. Also, empowerment is 
an important motivator for teachers. The genuine empowerment of groups of teachers in identifying, 
planning for, and implementing school-level strategies to reduce inequities can have a significant impact 
on the ways they think about themselves and their work. This process can often spread beyond the 
particular team itself, to others who are drawn into its activities (Ainscow et al., 2012). 


It seems but a short step from collaborative inquiry as an in-school strategy for improvement to the 
development of collaboration and networking across schools. Armstrong (2015) reports that many 
studies of such developments report improvements in areas such as professional development and 
career opportunities (Hill et al., 2012; West, 2010); sharing good practice and innovation (Stoll, 2015; 
Chapman et al., 2009); reductions and realignments in headteacher workload and organisational 
efficiency as a consequence of inter-school collaboration (Woods et al., 2006; Woods et al., 2013). 
However, he also reports that direct evidence of impact of student attainment is rather more limited, 
and, where it does exist, the picture is somewhat mixed. 


Some studies report no association between school involvement in inter-school collaborative activity 
and increases in student attainment (Woods et al., 2006; Sammons et al., 2007) whereas others 
suggest a possible association (Hutchings et al., 2012). The most notable of these is a large-scale 
research study with federations (schools brought together under shared governance and leadership 
arrangements), the findings from which suggest students attending certain types of federation 
outperformed a matched sample of their peers in non-federated schools in terms of their attainment 
(Chapman and Muijs, 2014). Nevertheless, widespread reporting of the positive influence on teachers’ 
attitudes and classroom practices that networking with colleagues generates does provide evidence of 
perceived impact, and it seems plausible that changes in teachers’ thinking and practices do arise from 
increased cooperation with colleagues from other schools (Woods et al., 2006). 


Wenger (1998), in putting forward his notion of ‘communities of practice’, also describes a process for 
the creation and transfer of knowledge within and across workplaces. ‘New’ knowledge acquired 
through this exchange can then be tested in practice in an individual’s own work contexts — though 
inevitably it will be modified as it is transferred and applied. In this way, ideas are moved around within 
the community, passing from practitioner to practitioner, and are continually modified and refined. Thus, 
it becomes possible for knowledge to be recycled around the community and returned to the originator 
— though transformed through the process. This may be significant for sustainability; we have seen that 
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networking models that essentially do little more than transfer knowledge from a ‘stronger’ to a ‘weaker’ 
partner have proved difficult to sustain. 


Further, our own research gives strong indications of how such processes can be initiated and 
managed (Ainscow and West, 2006). We have found the following practices foster successful 
networking: 


e the identification of common improvement priorities that are seen to be relevant to a wide 
range of stakeholders; 

e the presence of incentives that encourage key stakeholders to explore the possibility that 
collaboration will be in their own interests; 

e the development of a sense of collective responsibility for bringing about improvements in all 
the partner organisations; 

e headteachers and other senior staff in schools who are willing to drive collaboration forward 
but understand that the main knowledge transfer processes take place between teachers, 
who will therefore need time and opportunities; and 

e external help from credible consultants/advisers who provide process knowledge and 
guidance but do not act as ‘experts’. 


There is, then, some evidence relating to the characteristics of effective inter-school collaboration and, 
conversely, the main challenges to such activity. However, there is very little knowledge surrounding 
the change process and the development and maintenance of relationships when schools enter into 
collaboration. Negotiating change and brokering and nurturing new relationships are important factors 
in the sustainability of collaborative activity. There is more work to be done to develop the knowledge 
in this area. There is a lack of insight into the differential impact of inter-school collaboration and how 
different types of collaborative arrangements might vary in effectiveness, sustainability, and the kinds 
of impact they have. There is also very little evidence distinguishing between short- and long-term 
collaboration. There is a dearth of evidence within the literature relating to the means by which schools 
are selective about where, when, and in what ways to collaborate with other schools. Recent research 
into the emerging notion of a ‘school-led system’ has highlighted this (Sandals and Bryant, 2014) but 
more research is needed to explore the phenomenon in greater depth. 


Policy/Practice context 


It is clear that CtG is rooted in the belief that school-to-school collaboration has the potential both to 
drive and to spread strategies that help reduce the attainment gap. Focused as it is on an issue that 
was at the core of the London, Greater Manchester and Black Country Challenge programmes, and an 
approach that was used widely within these and subsequently borrowed by Schools’ Challenge Cymru, 
this is hardly surprising. Indeed, the attainment gap between disadvantaged pupils and their more 
advantaged peers has become an even greater cause for concern in recent years, particularly for 
schools outside London, where the attainment rate for children from poorer backgrounds continues to 
lag well behind that of their more fortunate schoolmates (Clifton and Cook, 2012). The explicit emphasis 
on the attainment levels and progress made by disadvantaged pupil groups within Ofsted inspections 
has obviously been one factor here, and regular feedback from PISA that the UK generally is making 
insufficient headway with this issue is another (Jerrim and Shure, 2016). 


However, other changes in the education landscape also argue that the activities that schools engage 
in as a result of participation in the CtG programme are increasingly in line with the restructuring of 
schooling currently taking place. For example, the continuing emphasis on school-to-school networking 
as a driver for improvement, and the parallel drive to establish multi-academy trusts (MAT) within which 
schools cooperate closely, providing both support and challenge, is a method of organising schools 
with which CtG sits very happily; in fact there is no reason why a MAT could not operate as a CtG 
cluster, with its schools nested in trios within. Manifestly, key elements of the CtG strategies for 
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improvement — improved use of data to track pupil progress, timely intervention based on monitoring 
data, supporting development in key curriculum areas through carefully planned activities outside the 
curriculum that can build confidence and skills, and using networking and collaboration to develop 
teachers’ practice and improve teaching quality — all match quite closely with current policy goals. 


Evaluation objectives 


During the first year of operation (pilot phase) the evaluation team worked closely with the CtG team to 
firm up expectations regarding outcomes and to agree parameters for evaluation. In the second year 
(2013-2014), the main year of delivery began with a larger group of new schools, alongside pilot 
schools continuing with the programme (both Lead and Accelerator schools). 


The focus of the evaluation throughout was on the impact of the programme on pupils and schools, and 
the research questions were: 


1. What is the impact of CtG on children’s academic attainment? 


a) Are there ‘differential gains’ in academic attainment for children eligible for 
free school meals (FSM)? 
b) Are there identifiable differences in attainment (raw differences) between 


Lead and Accelerator schools, either overall or for FSM children? 
2. What aspects of collaboration or other effective practice are most powerful in bringing 
about any improvements identified? 
3. What are the organisational conditions that are needed in order to make use of this 
approach? 
4. What are the barriers to these conditions, and how can they be overcome? 


The impact strand focuses on examining the outcomes in performance associated with participation in 
a school-to-school collaboration initiative, whereas the process strand investigates the nature of 
changes in school/classroom practices associated with this participation. The process strand can be 
further subdivided, into ‘illuminating cases’ and establishing an ‘implementation framework’. The former 
seeks to identify rich examples of individual practice of how pilot schools have engaged with CtG, 
whereas the latter seeks to establish indicators of common practices across extended schools in order 
to establish a theory of change describing how the programme is successfully implemented in schools. 
The findings of these two strands will then be combined in order to investigate both ‘what’ matters and 
‘how’ it matters in effective school-to-school partnerships. 


Ethical review 


All work involving primary data collection is subject to approval by the University of Manchester's 
Research Ethics Committee (UREC) in order to ensure that adequate consideration has been given to 
the ethical aspects of a research project, thus reducing the potential for harm and upset to the 
participants. 


CtG pilot schools were recruited by Challenge Partners (CP), the wider organisation that was 
responsible for the delivery of CtG, before the start of the evaluation. As only basic demographic and 
attainment data were utilised, these were provided through requests to the National Pupil Database. 
Initially case study schools were contacted and given copies of the case-study protocol, and letters 
were sent inviting participation in the evaluation (see Appendix 2). Letters pointed out that participation 
was voluntary and that no data collected would be reported in ways that allowed individual schools or 
staff members to be identified. The research team checked that staff members who were interviewed 
were aware of these documents, and also that they were happy to proceed with interviews on this basis. 
Individual permissions were sought from all participants interviewed. 
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In line with an initial agreement with the EEF, during the first year of the pilot the initial impressions of 
the evaluation team (based on attendance at workshops and interviews with staff in pilot schools) were 
shared with the CtG Co-ordinating Team, as was the interim evaluation report (containing both 
qualitative and quantitative analyses, the latter based on teacher records of pupil progress) that was 
produced at the beginning of year two. This was intended as formative feedback, though the Evaluation 
Team had no direct input to or influence on subsequent decisions made within or about the CtG 
programme. 


Project team 


The evaluation team was led by Professor Mel West, and comprised Professor Mel Ainscow, Dr Michael 
Wigelsworth, Dr Patricio Troncoso, with additional support from Dr Steve Courtney. 
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Methods 


Trial design 


As participating pilot schools had already been recruited and had begun to engage with CtG prior to the 
start of the evaluation, a randomised design was not possible. A matched comparison design was 
considered preferable to alternatives such as propensity scoring, as the intervention itself is diverse and 
relies upon a high degree of flexibility within individual schools. As such, there was insufficient data to 
model propensity scores on sufficient covariates or proxy data, risking an imbalanced or biased dataset. 
Instead, the evaluation utilised a matched-comparison design, using a number of school-level socio- 
demographic variables to enable comparison with similar schools not implementing CtG. As Challenge 
Partners recruited additional schools to CtG post piloting (see figure 1), these schools were 
subsequently included in the analyses of the later extension phase of the project (meaning the matching 
was updated to reflect the subsequent phase of the trial, but did not reflect any drop-out at this stage). 


The University of Manchester was provided with a list of all schools that enrolled in CtG (regardless of 
whether they dropped out at a later date) and their roles (either Pilot, Lead school or Accelerator 
school) by Challenge Partners. Schools were then matched into pairs as closely as possible to similar 
schools for which socio-demographic profile data was readily available 
(http://www.education.gov.uk/schools/performance), specifically: 


e % gap in attainment between non-FSM and FSM pupils (those eligible for free school meals 
at any time in the past six years) for pupils in previous cohort/pupils in cohort/based on school 
level data; 

e % of pupils eligible for free school meals at any time in the past six years; 

e % registered with Special Educational Needs or Disabilities for pupils in previous cohort/pupils 
in cohort/based on school level data; 

e size; and 

e local authority. 


For secondary schools: 

e % pupils in the previous year cohort who obtained A* - C GCSEs. 
For primary schools: 

e %pupils in the previous cohort who obtained level 4 on KS2. 


For each participating school, identifying all schools within the same and immediately surrounding local 
authority sourced a number of possible candidate matches (primary and secondary phases were 
completed separately). A final match was selected on the basis of having the most socio-demographic 
factors (as listed above) fall within 5% of the target school. Balance between trial arms (and 
comparisons to national averages) are shown in the Impact Evaluation section. 


Eligibility 


The school recruitment process was conducted by Challenge Partners, the organisation responsible for 
CtG. This involved direct marketing to schools and use of networks through existing schools 
participating in CtG. As evaluators, the University of Manchester was provided with a list of participating 
schools. This was conceptualised as two ‘waves’: first, the original pilot cohort (funded by the EEF); and 
second, an expanded number of schools that were recruited (their first year of participation funded by 
the EEF but the second coinciding with the launch of a school-based subscription model and utilising 
subsequent changes to the CtG programme, such as the introduction of facilitator schools). To be 
eligible, schools wishing to participate in CtG were also made aware of the evaluation parameters. No 
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primary data was collected for the reported analyses, utilising only anonymised basic socio- 
demographic indicators and national assessment data available from the National Pupil Database 
(NPD). As the intervention aimed to make broad, holistic changes in school practice, all pupils within a 
participating school were potentially eligible. 


Outcomes measures 


The primary outcome of the impact analysis was end of key stage national curriculum assessments 
(primary schools — Key Stage 2, secondary schools — Key Stage 4). This occurred at several time points 
dependent on phase of evaluation (see Figure 1). 


For primary schools, an average point score based on reading, maths, and writing assessments was 
used. For secondary schools, an average point score comprised a pupil’s best eight GCSE or GNVQ 
equivalent results. Average point scores were favoured over more direct comparisons of subject specific 
change (e.g. just Maths or English), given the whole-school approach of CtG (that positive improvement 
attributable to CtG was school, not class, based). In addition, these instruments mapped directly onto 
the intended goals of the CtG programme (academic attainment), and offered additional benefits in 
terms of external validity and zero additional data burden as a result of the evaluation (since end of key 
stage assessments are mandatory, and pupil-level data was requested from the NPD instead of being 
provided by schools). This was considered especially relevant given the nature of matched-comparison 
design. In addition, prior attainment data were requested (Key Stage 1 scores for primary, and Key 
Stage 2 for secondary). 


Sample size 


Schools had initially been recruited by Challenge Partners to participate in CtG prior to the 
commissioning of the evaluation. As an initial pilot, 30 schools (10 primary, 20 secondary) that 
participated in CtG completed the first two years of implementation (2012-2014). 


Challenge Partners recruited a total of 104 schools (65 primary, 39 secondary) schools in 2013. Forty- 
nine schools (29 primary, 20 secondary) remained active in 2014-2015 (end of the second year of the 
extension phase) (see Figure 1). All schools recruited in 2013 (i.e. 104) were included in the analysis. 


Table 2: Post-hoc power calculations* 


NES Tod YoYo) 
(Pupils) 
Primary schools 65 
0.103 0.175 
(extension phase) (7,096) 
Secondary schools 39 
0.102 0.27 
(extension phase) (13,945) 


*Power = 0.80, Alpha = .05. 
Analysis 


Outcome data was analysed using hierarchical linear modelling (also known as ‘multi-level modelling’) 
in MLWin Version 2.35 in view of the clustered and hierarchical nature of the datasets. 
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Two-level (school, pupil) models were constructed, with allocation status (CtG vs. comparison) and role 
(Lead / Accelerator) entered at the school level. Academic outcome data were entered at the child 
level, including both prior attainment (Key Stage 1 for primary, Key Stage 2 for secondary) and post- 
test (Key Stage 2 for primary, GCSE for secondary). We utilised both intention to treat (Gupta, 2011) 
and subgroup analyses where models were adjusted to include the relevant subgroup population of 
interest; the former provided an estimate of the overall impact of CtG on children’s academic attainment 
and the latter allowed us to identify any differential gains experienced as a result of school role (Lead / 
Accelerator) or by children eligible for FSM. Different stages of the programme are analysed: first, the 
impact of CtG on primary and secondary schools (extension phase) compared to comparison schools, 
including examination of any differential impact on pupils identified as eligible for free school meals 
(Table 6); second, examination of any differential impact in Lead or Accelerator schools (extension 
phase), including examination of any differential impact on pupils identified as eligible for free school 
meals (Table 7); third, a replication of the first analysis, using only schools who were identified by 
Challenge Partners as completing two years of the intervention (a proxy dosage measure) (Table 8); 
and fourth, the impact of CtG during the pilot phase, including examination of any differential impact on 
pupils identified as eligible for free school meals (Table 9). 


All continuous data were standardised (i.e. converted to z-scores) prior to analysis to facilitate 
comparison of effect size (ES) within and across models. All ES were then converted to Hedge’s g as 
per EEF specifications (Tymms, 2004) and 95% confidence intervals for the effect sizes were estimated 
as described by Fritz et al. (2012). 


MLWin by default uses all available data (e.g. for example it includes data for participants with and 
without complete data) using ‘maximum likelihood estimates’. This approach assumes missingness is 
at random (MAR) (this is where ‘missingness’ is assumed to depend on observed measurements). 
Crosstab examination of data showed some patterns of missing data (FSM vs. school role), meaning 
that there is a pattern to the missingness. However, it cannot be ascertained whether the pattern is 
directly related to factors pertinent to the study. For this reason, multilevel multiple imputation was 
performed using the software package Realcom-Impute (Carpenter et al., 2011) in combination with 
MLwiN. 


Missingness in the analyses of this report occurs in attainment variables only (key stage scores). A total 
of 893 (5.55%) of cases were missing for secondary schools and 391 (6.46%) of cases were missing 
for primary schools. Imputation models, a set of 10 multiply imputed for each analysis, were imported 
from Realcom-Impute back to MLwiN to re-run the multilevel models. Imputation was based on a 
chained equation using GCSE, KS2, FSM, CtG along with the school identifiers (to allow for variance 
between schools). This was based on the most complex model in the current analysis. No discernible 
difference was identified between the original and imputed datasets. The imputed dataset was retained 
for the main analysis, in order to retain statistical power. 


Implementation and process evaluation 


‘ 


Data collection for the process evaluation was detailed in the protocol thus: ‘...the process strand 
investigates the nature of changes in school/classroom practices associated with this participation. The 
process strand can be further subdivided, into ‘illuminating cases’ and the establishing of an 
‘implementation framework’. The former seeks to identify rich examples of individual practice of how 
pilot schools have engaged with CtG, whereas the latter seeks to establish indicators of common 
practices across extended schools in order to establish a theory of change describing how the 
programme is successfully implemented in schools.’ 


This approach involved a number of activities to generate data, including: 
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e meetings with programme teams to discuss the progress and development of the CtG 
programme implementation (these were front-end weighted, as we were asked to provide 
formative feedback to CtG during the first pilot year); 

e attendance at a number of workshops meetings, where the activities and discussions of staff 
from participating schools could be observed, including cluster review workshops (Workshop 
6) when participants themselves reflected on what they felt they had achieved during the 
previous year; 

e visits to participating schools to interview key staff members about their experiences in the 
CtG programme, and to compile short case-study accounts of the implementation and 
perceived impact within the school; 

e monitoring postings on the CtG website (which became operational towards the end of the 
first year of the project) where programme materials were available, and participating schools 
were also able to contribute materials. A number of schools produced short accounts of 
implementation and/or impact of CtG within their own schools, and there was also a facility for 
posting items for information and discussion; and 

e asurvey of CtG schools, carried out towards the end of the evaluation period. 


In the event, the local variations between schools and implementation strategies meant that the 
identification of a single ‘theory of change’ was not possible. Consequently, efforts were directed 
towards identifying common elements that schools associated with effective implementation and impact 
and common problems encountered through the data collection methods outlined above. In all around 
24 participating schools were visited, though not all were written up as case studies as it was not always 
possible to speak with enough staff members to ensure that a rounded view had been gathered, though 
insights from individuals were often interesting and did inform the development of themes and questions 
pursued within the case study schools. About a third of the cases were based on schools visited in both 
the first and second years of participation, and about the same proportion were based on schools that 
had or planned to withdraw after one year of participation. 


After the first year, when the workshops were centrally planned and, for the most part, delivered by a 
central team, it became more difficult to map the overall development as the participating schools 
became more spread geographically and the workshop programme was decentralised. Consequently, 
in the extension years, because the themed workshops followed a pattern that had been observed by 
members of the evaluation team in the pilot year, priority was given to attending workshop six, when 
local evaluations and reflections tended to be shared across the local clusters. These final workshops 
were observed in clusters operating in three different parts of the country in years two and three. 


Costs 


The CtG programme and the evaluation were commissioned before ‘cost per pupil’ became a standard 
metric within EEF reports — the process evaluation activities had been completed before EEF guidelines 
were published. In fact, even if detailed costing information had been available, it is hard to see how 
the CtG programme might have fitted easily into the EEF advice regarding the calculation of costs as 
implementation strategies and costs varied considerably between schools. 


Costs of implementation included the membership fee paid to CtG, the cost of release/cover for staff to 
attend workshops, and in many cases additional activities planned as part of the in-school 
implementation that were funded through Pupil Premium or other sources within the school budget. In 
the pilot phase in some cases there were also significant travel costs, as the workshop provision was 
not decentralised. However, neither CtG nor the evaluation team had made plans to collect this 
information. Inevitably the question of costs came up from time to time during visits to schools to compile 
case studies, and where cost was regarded as a significant factor in the school’s engagement with the 
programme this was recorded in the case studies; these references appear in the process evaluation. 
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However, achieving an accurate picture of non-financial costs would have been extremely difficult, even 
had it been planned for. This is because both the implementation model and the method of resourcing 
the model varied significantly between participating schools. Implementation involved three staff 
groups: teachers, leaders and a variety of non-teaching staff members (teaching assistants, librarians, 
administrators etc.) collectively referred to as ‘para-professionals’. The time commitment of the latter 
two groups could not be systematically identified, as generally, however much time was involved would 
either be outside the school day or not requiring cover. Teacher release requires more detailed 
planning, but though it sometimes required cover (which might be either ‘bought in’ or achieved through 
internal reallocation), significant amounts of teacher time too were outside the school day or fitted into 
non-teaching periods. A particularly difficult figure to estimate is the time spent on trio activities, as both 
the number of these, and the numbers and levels of staff involved varied between trios. However, it was 
clear from interviews with participants — both from schools that continued into a second year and also 
from many that did not — that teachers valued CtG as a vehicle for building leadership capacity. 


Timeline 


Pilot schools had initially been recruited by Challenge Partners to participate in CtG prior to the 
commissioning of the evaluation. The pilot phase began in 2012 with 39 schools (15 primary, 23 
secondary and 1 all-through). Three secondary, five primary and the one all through school dropped 
out of the pilot phase within the first year, meaning that 30 schools (20 secondary, 10 primary) from the 
pilot phase continued into a second year of implementation, joining the cohort of new schools recruited 
into the extension phase commencing in September 2013. 


Challenge Partners recruited a total of 104 new schools (65 primary, 39 secondary) in 2013, the first 
year of the extension phase. Of these, 36 primary schools and 19 secondary schools had dropped out 
of the programme before the beginning of the second year of the extension phase, meaning 49 schools 
(20 secondary, 29 primary) continued into the second year of the extension phase, where they were 
again joined by additional recruits. Drop out across phases is shown in Figure 1. 


For both the pilot and extension phase attrition figures were reported by Challenge Partners. 
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Figure 1: Timeline of CtG and analysis 


2013-2014 
Extension phase 


ITT Analysis 
(started in 2013-2014) 


Pilot evaluation i 
(started in 2012-2013) | 


65 schools 


10 primary schools 
(7,096 pupils) 


(1,605 pupils) 


39 schools 


20 secondary schools 
(13,945 pupils) 


(7,269 pupils) 


2014-2015 
Extension phase 


Proxy dosage analysis 
(remained in 2014-2015) 


29 remain 
(3,577 pupils) 


20 remain 
(7,912 pupils) 


End of Key Stage 2 (primary) and Key Stage 4 (secondary) assessments were requested from the NPD 
for the end of the pilot (2013) and extension (2015) phases for all pupils in participating schools and 
their matched comparisons (see ‘Trial design’ section for details of the matching procedure). 


The following analysis reports on the impact of CtG for the extension phase of evaluation. 
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Impact evaluation 


This section outlines the outcome analysis for the CtG programme for primary and secondary schools, 
and focuses on the final extension phase of the evaluation. It is organised into five sections. 


First, there is an examination of the school characteristics in both arms of the trial (CtG vs. comparison) 
as well as national averages provided as a reference point with which to assess the representativeness 
of sample (Tables 3 and 4). 


Second, there is an analysis of the impact of CtG on primary and secondary schools (extension phase, 
n=104), compared to comparison schools, including examination of any differential impact on pupils 
identified as eligible for free school meals (Table 5). 


Third, there is a follow-up of the preceding analysis in order to identify any differential impact in Lead or 
Accelerator schools (extension phase, n=104), including examination of any differential impact on pupils 
identified as eligible for free school meals (Table 6). 


Fourth, there is a replication of the first analysis, using the subset of schools (n=49) who were identified 
by Challenge Partners as completing two years of the intervention (a proxy dosage measure) (Table 
7). 


Fifth shows the impact of CtG during the pilot phase (n=30), including examination of any differential 
impact on pupils identified as eligible for free school meals (Table 8). Full details of each analytical 
model are reported in the appendices. 


School characteristics 


Tables 3 and 4 summarise the characteristics of Lead, Accelerator, and comparison schools and pupils, 
with national averages provided as a reference point with which to assess the representativeness of 
the sample. 


Table 3: School and pupil sample characteristics and national averages: Primary schools (N=65) 


School/Pupil National CtGschools Comparisonschools _ Balance at 
characteristic average mean mean baseline 
(J 3) (SD) (ozo) nt-Ya emo) ) 
; 409.26 408.61 
School size 1 : 
ae (161.81) (145.47) anki 
: : * 14.34 15.69 
Gap in attainment 2 
e 316 (10.81) (10.92) vale 
Average point score at 07 28.87 28.8 0.041 
KS2 (current cohort) (1.56) (1.53) : 
ini 24.79 25.14 
% FSM eligible 1 
° q 168 (13.75) (13.08) ones 
10.8 8.65 
% SEN 1 : 
sie ie (11.64) (4.21) O:238 
39.87 46.83 
% EAL 1 
‘i ie (28.44) (27.3) area? 
: 52.06 50.06 
% Male pupils 1 c 
0 pup 50 (5.65) (5.1) 0.369 
KS1 score (prior 14.89 14.91 
attainment) 16 (1.24) (1.14) one! 


‘Percentage difference between those eligible / not eligible for FSM achieving level 4 or above in reading, writing 
and mathematics at Key Stage 2, as measured by the disadvantaged pupils attainment gap index. 
‘Department for Education (2015a), 7Department for Education (2015b). 
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CtG schools show a close similarity to national averages on a number of socio-demographic variables, 
specifically average attainment, gender balance, and prior attainment of pupils. However, CtG schools 
have (unsurprisingly) a higher than average gap in attainment. In addition, CtG schools have a higher 
than average percentage of pupils eligible for free school meals, a higher percentage of pupils with 
English as an additional language, and a lower percentage of pupils identified with special educational 
needs. 


Balance between CtG and comparison schools is shown to be very close, with most socio-demographic 
variables showing less than a tenth of standard deviation difference. In the case of % EAL, % SEN, and 
% Male, the nature of the calculation should be taken into account. As this is based on a percentage 
score, differences in Cohen’s d can be magnified. There is approximately a 7% difference in the number 
of children with EAL. 


Table 4: School and pupil sample characteristics and national averages: Secondary schools 


(N=39) 


School/Pupil National CtGschools Comparison schools | Balance at 
characteristic EW -16-(e[=) mean mean baseline 
((}3)) (J) (Coxe) nt -Yae— ep) 
School size ; 1033.46 1186.05 
ce (321.04) (357.65) one 
Gap in attainment* 2 24.15 21.92 E 
p 3.74 (9.8) (9.43) 0.23 
Average point score at 320.32 321.99 
KS4 (current cohort) oueet (28.57) (29.5) one 
% FSM eligible 1 16.26 16.63 
0 g 14.9 (8.94) (9.6) 0.04 
% SEN F 8.25 7.95 : 
o 18.2 (3.75) (3.37) 0.083 
19.7 23.52 
% EAL 1 
: te (24.58) (25.71) pee 
about gad a (21.8) (21.8) ae 
*Percentage difference between those eligible / not eligible achieving at least 5 A*Y-C GCSE English and Maths at 


Key Stage 4. 
'Department for Education (2015a), 7Department for Education (2015b). 


As with primary schools, the secondary sample also shows a higher percentage of FSM eligibility and 
a higher than average gap in attainment, in comparison to the national average. Conversely, % SEN 
is lower in the current sample, compared to national averages, though % EAL is higher. CtG and 
comparison groups show good (though not perfect) balance at baseline, as although there are 
differences across most variables, these are mostly small, as indicated by the effect size. Although 
Cohen's d is reported to be high for school size and % Male, this is due to the calculation being made 
from a percentage score. The actual number of pupils is very small. 
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Impact of CtG on primary and secondary schools 


Table 5: The impact of CtG on pupils’ academic attainment (intent to treat) 
Post-test means* Effect size 


Outcome e3(€) Comparison 


| Mean N Mean Hedge’s Months’ Statistical 

(missing) (95%Cl) (missing) (95% Cl) g progress _ significance 
Es (95% 
oi) aa 


Primary schools (N=65) 


Key Stage 2 Bak . p=0.993 
score : non sig. 


Key Stage 2 
score — if FSM 
eligible 


Secondary schools (N=39) 


Key Stage 4 13131 aes p=0.926 
score (814) . non sig. 


Key Stage 4 
score — if FSM 
eligible 


“These correspond to predicted means based on multiply-imputed multilevel models. Original scores have been 
standardised for ease of comparison across models. 

“ The number of missing corresponds here to the cases dropped listwise in that category when KS2 or KS1 scores 
are missing. 

This is the effect size of CTG participants as opposed to the comparison schools. 


As can be seen from Table 5, the overall trend is that CtG did not impact upon pupils’ attainment. This 
finding is consistent across both primary and secondary schools, although there is a small gain in 
months’ progress (two months) for FSM eligible primary school pupils. FSM eligible pupils in Key Stage 
4 show a statistically significant decline in two months’ progress when compared to comparison schools. 


Examination of differential impact in Lead or Accelerator schools 


The following analysis considers the effect for particular roles within CtG. For instance, schools 
identified as ‘Lead’ should produce greater results when compared with all other schools in the study 
(consistent with programme theory as drivers of good practice that should be (a) disseminated to 
Accelerator schools, and (b) absent from comparison schools). 


In addition, the analysis considers a series of interactions. This examines whether there is an increased 
effect between two factors; for example, whether being in a Lead school has a greater effect on pupils 
eligible for free school meals. This is denoted in the table as an asterisk between the two variables (e.g. 
FSM* Lead). 


Mean values have been predicted using the multiply-imputed datasets. They are the means for 
particular groups (comparison or participant, FSM or non-FSM, etc.) while controlling for the other 
variables in the model. 
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Table 6: Differential impact in Lead or Accelerator schools 
Outcome Post-test means* Effect size 


N Mean Hedge’s g Months’ Statistical 
(missing) (95% Cl) (95% Cl) progress ite Taliicer-lalers. 


Primary schools (N=65) 


CtGLead | 830 (58) 0.18 -0.07 
(vs. (0.13; 0.23) (-0.13; 0.002) 
comparator 
schools) 
CtG 5766 (442) 0.031 0.006 
Accelerator (0.011; 0.051) (-0.03; 0.04) 
(vs. 
comparator 
schools) 
| a 


FSM 2810 -0.31 -0.2 
(vs. not (150 (-0.34; -0.28) (-0.24; -0.16) 
FSM - p<0.01 
comparator 
schools) 
(9) (-0.25; -0.036) (0.021; 0.324) pais 
Accelerator (69) -0.36; -0.28) (0.035; 0.147) as 


Secondary schools (N=39) 


CtG Lead 5522 0.123 ; 
(vs. (293) (0.12; 0.13) 
comparator 
schools) 
CtG 7609 -0.1 -0.09 
Accelerator (521) (-0.11; -0.094) (-0.12; -0.07) 
(vs. p=0.256 
comparator 
schools) 


p=0.171 


FSM 3835 -0.49 -0.53 
(vs. not (217) (-0.494; -0.481) (-0.57; -0.5) 
FSM — 
comparator 
schools) 


FSM* Lead 681 -0.41 -0.11 

Accelerator (59) (-0.625; -0.6) (-0.14; 0.02) 7 
“These correspond to predicted means based on multiply-imputed multilevel models. Original scores have been 
standardised for ease of comparison across models. 


p<0.001 


-1 
-3 
2 
1 
2 
-1 
-7 
-2 
1 


In examination of the different roles within CtG, there can be seen there is no significant differential 
impact of CtG for either Lead or Accelerator primary schools. There are small (1-2 months) marginally 
non-significant gains shown for both FSM eligible pupils in Lead primary schools (FSM* Lead) and FSM 
eligible pupils in Accelerator primary schools (FSM* Accelerator). An associated finding is that across 
the whole study (Lead, Accelerator, and comparison) FSM eligible pupils show a decline in progress. 
This does not necessarily reflect a decline, but instead the recognized ‘attainment gap’ for this group, 
when compared to pupils otherwise not eligible for FSM, i.e. that there is discrepancy between non- 
FSM and FSM pupils and this intervention does not narrow the gap. This result should be compared 
with earlier tables to establish the effect of CtG on FSM eligible pupils. 
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Regarding secondary schools, it can be seen that no significant gains were made for Lead or 
Accelerator schools. A significant negative effect is noted for the interaction between FSM eligibility and 
Lead (FSM* Lead), indicating that there is no improvement in attainment for FSM eligible pupils 
specifically within Lead schools. A similar result is shown for FSM eligible pupils in Accelerator schools 
(FSM* Accelerator). However, as above, these results may represent a ‘lessening of the gap’, rather 
than a directly negative effect. 


In summary, there is evidence to indicate that primary Lead and Accelerator schools may be slightly 
reducing the attainment gap for FSM eligible pupils, but negative findings in the secondary schools do 
not support the same conclusion. As pupils not eligible for FSM are not improving either, this implies a 
lack of overall effect for CtG. 


The next analysis considers whether there is evidence to suggest whether this is a result of lack of 
engagement with the programme by comparing results for schools that were reported to have actively 
maintained CtG for two years. 


Proxy dosage (analysis of schools identified as completing two years of CtG) 


This analysis uses a subset of the data analysed in Table 5, specifically schools identified by Challenge 
Partners as completing two years of the intervention (and their corresponding matched comparison 
schools). 


Table 7: The impact of CtG on pupils’ academic attainment (proxy dosage: for those schools 
identified by Challenge Partners as completing two years of the intervention) 


Post-test means* Effect size 


Outcome CtG Comparison 


N Mean N Mean Hedge’s Months’ Statistical 
(missing)** (95% (missing) (95% Cl) g progress significance 
)) (95% 
cl)*** 


Primary schools (N=29) 


0.12 0.07 -0.02 
— 2 3333 (244) (0.10; 2325 (147) (0.04; (-0.07; a 
oe) 0.11) 0.04 9. 


Key Stage 2 -0.34 -0.320 -0.34 
score -—ifFSM | 589 (86) | (-0.41; (27) (-0.39; es s 
eligible o 28) yi 26) 9- 


Secondary schools (N=20) 


-0.01 0.01 
Key Stage 4 . ; . p=0.95 
7466 (446) | (-0.02; | 7728 (447) | (0.00; (-0.04; 
= reso) 0.01 rien 0.01) 02) a 


Key Stage 4 -0.50 -0.46 -0.09 


946 115 =0.07 


score — if FSM : 
-0.44) -0.02) has 


eligible 


* These correspond to predicted means based on multiply-imputed multilevel models. Original scores have been 
standardised for ease of comparison across models. 

*“ The number of missing corresponds here to the cases dropped listwise in that category when KS2 or KS1 scores 
are missing. 

*** This is the effect size of CtG participants as opposed to the comparison schools. 


Results show little improvement across key stages and FSM eligibility. Although a non-significant finding 
is shown for Key Stage 2 pupils who are eligible for FSM, the effect is equivalent to one month’s 
progress. 
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In juxtaposition with previous analyses, results may show a ‘reduction of decline’ in which failure to 
‘make progress’ is interpreted as a halting of an otherwise widening gap. This is consistent with previous 
analyses, as results in this analysis are an improvement when compared to Table 6 (which does not 
consider ‘proxy dosage’, i.e. maintenance of CtG over two years). This may indicate the need to 
continue CtG in order to stem an otherwise widening gap. 


Examination of the CtG pilot phase 


Table 8: The impact of CtG on primary and secondary schools (pilot phase) 
Post-test means Effect size 


Outcome CtG Comparison 


| Mean | Mean Hedge’s Months’ Statistical 
(missing) (95% (ralessiare)) (95% g progress _ significance 
(e})) (o})) ‘CEy~mel)) 


Primary schools (N = 10) 


Key Stage -0.04 (- -0.05 0.03 
2 score 1012 0.09: he (0.10: (-0.07: 1 p=0.85 
(53) 0.01) (35) 0.00) 0.12) 
pipe oy -0.54 (- 0.46 0.09 
eeetodr | Nee 0.65; be 0.60; | (-0.06; { =0.49 
anid (12) 0.42) (9) : ve Bee _ 
eligible ae) O32) 25) 
Secondary schools (N=20) 
Key Stage 0.05 -0.21 0.38 
4 score iced (0.04: 5455 (467) (-0.19; (0.34: 5 p=0.06 
0.06) 0.23) | 0.41) 
Key Stage ; ‘ ; 
ype 0.30 ( Sep 0.65 0.09 
1464 (127) 0.27: (0.68; (0.04: { p=0.049 
aiid 0.33) (76) 0.62) 0.14) 
eligible . : . 


In brief, consideration of the pilot phase results findings from Table 9 suggests that, for primary schools, 
small progress in attainment is made for both FSM eligible pupils and the wider cohort. Consideration 
of the effect size indicates that progress for FSM eligible pupils (g = 0.09) is more than that of the wider 
cohort (g = 0.03), reducing the attainment gap. However, effects are to the second decimal place, 
indicating a very small effect, which has also failed to reach statistical significance. Although results for 
secondary schools show close to statistical significance (both marginally significant and marginally non- 
significant) and larger effects, the majority of gains are seen for the wider cohort (g = 0.38) in 
comparison to FSM eligible pupils (g = 0.09), in effect widening the attainment gap. 
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Costs 


It would be possible to apply the EEF proposed formula for calculating the financial cost per pupil if we 
assume that the total number of pupils in the school ‘benefit’ from what is intended to become a whole- 
school intervention, though it begins with a cohort of 15 pupils. On this basis, using the current annual 
fees of £4,975 for primary schools and £8,500 for secondary schools, the cost per pupil per year of the 
intervention is as follows: 


e Primary schools: between £5.65 (880 pupils) and £33.16 (150 pupils) 
e Secondary schools: between £5 (1700 pupils) and £13.82 (615 pupils) 


Table 9: Costs per pupil over time in average primary school (based on average of 267 pupils 
per school)' 


Cumulative cost 


Cost per year over 3 


Cumulative cost per Average cost per pupil 


years Yeq afore) m6) per pupil (£) per year (£) 
Year 1 £4,975 £18.63 £18.63 
Year 2 £9,950 £37.27 £18.63 
Year 3 £14,925 £55.90 £18.63 


Table 10: Costs per pupil over time: secondary school (based on average of 939 pupils per 


school)? 
Cost per year over 3 Cumulative cost Cumulative cost Average cost per pupil 
years per school (£) per pupil (£) per year (£) 
Year 1 £8,500 £9.05 £9.05 
Year 2 £17,000 £18.10 £9.05 
Year 3 £25,500 £27.16 £9.05 


1 https:/Awww.gov.uk/government/statistics/schools-pupils-and-their-characteristics-january-2016 
2 https://www.gov.uk/government/statistics/schools-pupils-and-their-characteristics-january-2016 
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Process evaluation 


Note: The data informing the process evaluation was for the most part collected during the pilot and 
initial extension phases. We are aware that there have been a number of modifications since that time, 
some of which directly address issues raised by these accounts from the early participants. Readers 
wishing to know the detail of the current programme are advised to contact CtG directly or view current 
descriptions of the programme on the CtG website. 


As we have noted above, the process evaluation was framed around a series of research questions: 


1. What are the most important features of the CtG approach? 

2. What are the organisational conditions that are needed in order to make use of this 
approach? 

3. What are the barriers to these conditions, and how can they be overcome? 


The first question is the focus of the quantitative sections of this report; here, we look at key aspects of 
the implementation model, drawing on data gathered through a series of case studies of implementation 
in a sample of schools, and through a survey of participating schools that was administered after the 
end of the second year of the extension programme. A total of 17 case studies were compiled, of which 
about half were based on schools with one year’s experience of the programme and half were based 
on schools that completed a second year. The survey was open to all schools that had participated in 
year two of the extension, whether this was the first or second year of involvement. As we note when 
reporting the survey results, the response rate (around 25%) was not high, and perhaps we can assume 
that schools that did respond were those where involvement in CtG had generated the strongest views. 


How do schools, clusters, and programmes operate? 


As is clear from the programme description, networking with partner schools is central to the 
implementation model of Challenge the Gap. Of course, there have been a number of improvement 
initiatives over the years, both local and national, that have placed emphasis on schools working 
together in order to deliver the desired outcomes. However, some have argued that while schools have 
shown a willingness to cooperate when this has been necessary to obtain resources, genuine 
collaboration has not always ensued once resources have been secured. While this may have been a 
consideration in the pilot phase of CtG, when participating schools received funding support from EEF, 
the major extension phase was largely self-funded. Consequently, it seems fair to assume that the 
majority of schools that took part did so because the goals and structure of the programme seemed 
directly relevant to their own contexts, and were at least willing to contemplate that networking — loosely 
with a cluster of schools and closely with two partner schools within a trio — would be an effective 
improvement strategy. Below we consider what seem to be strengths and limitations of this approach, 
which is integral to implementation of the programme. 


As noted above, the literature relating to effective networking suggests that a number of factors are 
present in successful networks. Fundamental among these are common priorities and appropriate 
incentives. Self-evidently, the CtG schools shared a common goal: the need to reduce the gap 
between overall attainment levels and the performance of pupil premium pupils. However, there is 
evidence in the case studies that what was shared here went beyond simply securing improved pupil 
outcomes, and extended to improving pupil engagement and improving the quality of learning 
experiences provided. Thus one teacher reported ‘/t went from just sharing good practice to actually 
using it to develop all of our practice...’, while another commented that the opportunity to see the 
problems pupil premium pupils faced in different school contexts had resulted in a much clearer 
understanding of ‘that particular group of vulnerable children and their particular sets of circumstances, 
which | think has really benefited them. The opportunities that we gave to them, and the things that they 
experienced that — maybe — they wouldn’t have had, have really enriched the curriculum for them’. Key 
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elements of the CtG programme are the specific strategies to boost progress in numeracy and literacy, 
but there is a general recognition that this is not simply a matter of more effective teaching and learning. 
‘We are emphasizing the building of cultural capital, this is not just about accelerating progress, it is 
about raising aspirations and expectations, increasing self-awareness and life skills. We need to extend 
the horizons for these children.’ 


Similarly, for many the opportunities to work with colleagues from other schools facing similar 
problems and to share practice were important incentives, encouraging cooperation between 
school leaders and (especially) teacher commitment. ‘That’s what really struck me, the collaboration 
aspect of the programme and the opportunities for working together across boundaries — that’s what 
really, really made it’, one teacher reported, while another summarised the wider benefits the 
programme offered to the school thus: ‘The opportunity to talk to other professionals in other schools 
has been invaluable... when you meet other people and hear about the context of their school, what 
they face and what their difficulties are, it helps you to really evaluate the strengths of your own school, 
but also the areas that could be improved further, and | found that really helpful.’ Another participant 
described how working with other schools had ‘added momentum to the school’s efforts. ‘We have had 
tremendous support, it’s like a big family, | feel very committed. What has been brilliant is to be 
supported every step of the way.’ 


But enlightened self-interest was clearly a strong factor too: ‘... we don’t have all the answers, and we 
know it. So that was part of the appeal. But so was working with other schools on the same problem, 
especially the cluster lead school... which had been involved in this programme for some time’. 


Research evidence from previous studies (West 2010, Armstrong 2015) also suggests that a sense of 
collective responsibility for bringing about the desired improvements in all the partner schools 
characterises successful networking arrangements. Again, we can find ample evidence of such sharing 
within the school case studies. One school reported how, at the start of each year, staff members from 
each of the trio of schools had met to discuss priorities and to plan the areas they would focus on for 
the following school year within CtG. This way they ensured that there was a level of consistency across 
the trio in terms of what they were working on, and also allowed more meaningful exchanges of data 
between the schools in terms of overall impact on pupil learning and outcomes in all schools. It is clear 
that this trio felt that supporting the efforts of their colleagues was as important as moving things forward 
in their own schools. In another trio, a similar process for ensuring that schools understood the detail of 
what each other was hoping to achieve and contributing to one another’s plans was described: 


‘Initially, different schools were using different year groups so we did an overall plan of what we wanted 
to see throughout the three schools and how we were going to work together to do something similar, 
which was good and has meant the data from all three schools is more meaningful. We've also created 
Planning Progress Outcomes (PPO) to identify what the leaders, teachers, and paraprofessionals are 
aiming to achieve in each case... and we’ve all worked on this document together.’ 


This commitment to ensure progress for all often resulted not just in joint planning of common activities, 
but in schools bringing together pupils to carry these activities out together. There are many examples 
of children from different schools going on trips together, or visiting one another’s schools, including 
some examples of cross-phase visiting, with primary pupils visiting secondary schools and, rather more 
unusual, secondary pupils visiting primary schools. Generally, participating schools speak very 
positively of such activities, pointing out that activities take pupils beyond the boundaries of their 
own schools and constitute important events for the pupils involved, building confidence as well 
as offering useful learning opportunities. 


A fourth finding that emerges from research into networking between schools relates to the role played 
by senior leaders. Above, we report that this research (Armstrong, 2015) indicates that headteachers 
(and other senior staff) in schools are instrumental in initiating and driving forward cross-school 
collaboration, though they also need to understand that the important knowledge transfer processes 
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take place between teachers, who therefore need both time and opportunities for this to happen. We 
can find evidence within CtG that school leaders appreciated both of these aspects of their role. Thus, 
as one case study relates: 


‘The decision that the school would join the Challenge the Gap programme was taken by the 
heaateacher. It seemed likely that the stimulus to join had been existing relationships between a trio of 
headteachers, who had similar concerns and had decided jointly that this might offer a potential 
improvement strategy for all their schools.’ 


Once this decision had been taken, however, it was how the programme was managed in schools that 
staff commented on most often. Unsurprisingly several respondents linked the importance of having 
sufficient time and resources available for the programme activities directly to its success, and 
typical comments here included: 


‘Time to plan, funding for meetings and activities’ 

‘Time, more time! Good paras who are willing to accept the challenge themselves’ and 
‘Ensuring enough time for discussion, planning and implementation of interventions is allocated and 
protected’ 

‘Staff working together purposefully. Time to plan. Regular reviews of progress’ 


Our impression here was that these teachers appreciated the extended, whole-school commitment to 
CtG, which had protected time to work on the programme. Such protection, it seemed, has not always 
been a feature of either school level interventions or cross-school working arrangements, and here the 
fact that it was available and was extended to the teaching assistants working on the programme were 
seen as crucial determinants of impact. 


In one school we heard that while senior leaders were keen to establish expectations for the trio that 
were both realistic and manageable, they were acutely aware of the additional workload for teachers 
and paraprofessionals associated with inter-school collaboration. Over the course of the programme 
they have ensured that staff members have been able to meet on a regular basis as a trio, while the 
CtG pupil cohorts have also had opportunities to meet with their counterparts from the other schools. 
One teacher explained: ‘We've had three visits to each other's schools or ‘host days’, and then probably 
another three planning sessions together.’ But a teacher from another case study school described 
what it entailed in more detail, stressing that development programmes need, above all, ‘Capacity. 
Projects like this never work if your teachers and support staff don’t have the time to really devote to it 
to make it happen. The second thing for the teachers was that they needed to have enough authority 
within their department to take things forward.’ The provision of ‘time’ is certainly one important 
component of capacity, but this teacher identified an equally important resource: the authority to make 
decisions needed to move the project forward. There is no doubt that the time available for development 
work in schools is limited, but whatever time is available will be used more productively if the appropriate 
authority to act has been delegated to those directly involved. 


Previous research findings (Ainscow, 2015) have also pointed to a key role often played by external 
consultants or ‘experts’ within school-to-school collaborations, suggesting that some form of knowledge 
brokering between parties may be helpful. Within CtG it seems that this function was at least partially 
met through the workshops, in two ways. First, even when facilitated by local ‘hubs’, these training 
events drew substantially on centrally produced resources. In this way, a steady stream of ideas and 
materials — for the most part drawn from research into strategies to accelerate the learning of 
disadvantaged groups — was fed into deliberations and discussions of the school trios. Thus, as one 
school lead reported: ‘We’ve been able to share a body of knowledge about strategies to work with 
disadvantaged youngsters, both within the classroom and outside the classroom with parents and also, 
from a leadership perspective, how to have difficult conversations with teachers who are perhaps not 
focusing enough attention on disadvantaged children.’ 
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A second source of ‘advice’ available at these workshops was the reports from other trios within the 
cluster, where schools regularly reported the problems they were facing, the strategies they were using 
to offset these, and what they were learning from this experience. This too seemed to be generally 
appreciated, for example: 


‘... (at that workshop) there was thinking about resilience and resilient learners and they talked about 
an intervention at the lead school there, that we've basically pretty much stolen and we're trying to put 
into place now, which is having all of the students... giving a talk during their form time.’ 


It seems clear from the above that, to a large extent, the experiences reported by schools taking part in 
CtG reinforces previous research findings regarding the conditions that lead to effective networking: the 
majority of the participants believe that working closely with colleagues from other schools has 
added an additional dimension to their efforts. 


Several strengths of the programme have been mentioned above. These include structured 
arrangements and support for the networking activities, with some evidence that these have become 
stronger from one year to another as the programme has moved to more local coordination and 
facilitation via the local ‘hubs’. Another important feature is the explicit requirement that 
networking involves teachers and ‘paraprofessionals’ directly, rather than being something that 
takes place between senior leaders and is then ‘cascaded’ into the school. Time and again during 
school visits, and also prominent in questionnaire returns, the important role played by teaching 
assistants was mentioned. This was spoken about both by teaching staff, keen to record the vital 
contributions these colleagues had made to the programme implementation: 


‘...the TAs understand how to best support PP pupils in all classes and they have developed strong 
relationships with the pupils in the groups’ 


and also by the teaching assistants themselves, who reported that they felt much more involved in 
planning for and supporting pupil progress. A further strength has been that the majority of the 
participating schools have — perhaps because they have conducted the programme as an exercise in 
collaborative inquiry — been eager both to apply what they have learned rapidly to other pupil groups 
within the school and also to transfer this between schools. 


However, there are other strengths emerging that are less well described in the literature. For example, 
the practical collaborations between schools have often extended to the target pupil groups, offering 
them new experiences and through cross-school contact, deepening the feeling of collective 
responsibility for achieving the intended outcomes for all involved. The joint planning activities have 
been extensive, despite in many cases trio schools developing different points of focus and different 
strategies for achieving these. Perhaps because of this, relationships and trust between teachers within 
the school trios seem especially strong in some instances, and several have already extended their 
collaborations beyond the scope of the project or beyond formal engagement in it. Perhaps this is little 
more than shared success leading to further collaboration, but there is no doubt that Challenge the Gap 
seems to have struck a chord among participating staff members, teachers, and teaching assistants 
alike, that has galvanised their efforts. 


This highlights the differences in programme ‘impact’ that has been recorded between the qualitative 
and quantitative strands of this evaluation. While the attainment data covering the evaluation period 
suggests that the impact of the programme was slight, this is based on a limited measure that perhaps 
does not capture aspects of pupil ‘progress’ in the broader sense, for example improved engagement. 
The case studies (a summary of key observations from case study school participants is available in 
Appendix 3) and the findings from the survey of participating schools conducted at the end of 2015 both 
offer strong evidence that those involved — school leaders, teachers, and para-professionals — all had 
extremely positive views of the programme, and of their own school’s development through and benefits 
from participation. While these views strongly feature professional development and school-to-school 
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cooperation activities, there were also many instances where teachers referred to visible improvements 
in pupils’ progress, though these were often expressed in terms of changes in attitudes, engagement 
or attendance, rather than demonstrated in measured attainment. In fact, many of those involved clearly 
believed that pupil progress had also been accelerated, at least during the early period of their 
participation in the programme. Interestingly, this echoes the findings from the Evaluation of the City 
Challenges Programme, which reported that: 


‘The majority of schools involved in this evaluation reported that their strategies to tackle attainment 
gaps were successful. The programme was successful through its reach in terms of raising awareness 
of FSM gaps and the systematic disadvantage that some students are facing.’ (Hutchings et al., 2012) 


The same report points out that, in fact: 


‘..among pupils in London and the Black Country Primary schools with CtG funding, the FSM 
attainment gap narrowed more than it did among pupils in schools not known to have funding. However, 
this was not the case in Greater Manchester, where there was little difference between funded and 
unfunded schools. However, in Secondary schools, the FSM attainment gap widened among pupils in 
CtG-funded schools in each area.’ (Hutchings et al., 2012) 


So, we see that here too, there was a clear perception among those involved that the programme was 
accelerating the progress of targeted disadvantaged pupils, though no clear evidence that this led to a 
narrowing of the attainment gap. A possible explanation for this may lie in the comments made by a 
group of teachers interviewed in one of the case study schools: 


‘When asked whether they thought the programme had actually ‘narrowed the gap’, those involved felt 
this was the wrong question. One problem here was the change in national curriculum progress 
monitoring, and the fact that there was now no ‘levelling’. However, the bigger issue related to the 
usefulness of reducing the current in-school gap. The important thing was to improve attainment for all 
pupils. This was particularly important in a school where so many of the pupils could be considered 
disadvantaged, and the dividing line between the target group and the rest of the cohort was both small 
and arbitrary’ (Village Primary School case study) 


It is, of course, possible to increase attainment without narrowing the gap. If the approaches developed 
and implemented with the CtG programme target cohort did accelerate progress, but were rapidly rolled 
out across classes and year groups, then perhaps schools would see similar acceleration across the 
board. However, if this perception is accurate, we would expect to see supporting evidence in the 
quantitative evaluation. 


Finally, we should note the profound impact that the programme has had on some teachers from 
single-form entry primary schools, who reported that the trios provided them with the first 
opportunity available to them in their professional lives to plan teaching jointly with an 
experienced colleague. Of course, they had previously opportunities to lesson plan with trainees and 
sometimes helped NQTs, but this was seen as a wholly different experience from cooperative planning 
with an experienced colleague, and in some cases had become a form of cross-school lesson study. 


What arrangements at school level best support the CtG implementation model? 


The case studies and the survey responses tell us quite a lot about the way CtG was implemented, the 
perceptions of those directly involved in CtG, and their feelings about its usefulness as an intervention 
to improve outcomes for disadvantaged pupils. On the one hand, we can see that there are a number 
of common elements that seem to run across most participating schools, and these include the nature 
and size of the target group, the willingness to collaborate both within and across schools to support 
this group, deliberate and regular monitoring of impact, and commitment to sustain and spread those 
aspects of the intervention that seem to impact most clearly on pupils’ progress. At the same time, there 
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are many variations within the bundles of activities these interventions comprise. Of course, there are 
similar activities — cultural visits and experiences, activities designed to develop confidence, or self— 
esteem, or social skills — but these are mixed in different ways, reflecting contexts and opportunities. 


Broadly the elements can be seen as related to the practices and dispositions of the staff involved, the 
nature of the target cohort, and the opportunities available for extension activities. 


Staff dispositions 


We have noted above that the ‘advice’ from CtG implies that significant numbers of staff will be directly 
involved in the training, implementation and school trio activities; in some cases, as many staff as there 
are students in the target cohort. Though we were only able to visit a sample of the schools involved, 
school visits and attendance at the training workshops suggested that during the pilot year most 
participating schools tried to follow the programme model closely, though as the programme has moved 
into subsequent years there do appear to have been smaller numbers of staff involved in the 
participating schools — as we heard from some interviewees and was noted by one of the questionnaire 
respondents, attendance at workshops has not always been maintained, and as case study accounts 
show, the cross-school contact and activity has also been reduced in some instances. However, the 
size of the staff cohort recommended does underline the longer term aim of CtG, and, somewhat 
paradoxically, its concern to secure sustainability. The target cohort may be the initial beneficiaries of 
the intervention, but from the outset the goal of the programme is not only to develop ideas and 
strategies for accelerating the progress of disadvantaged pupil groups, but to ensure that within each 
school there is team with the capacity to replicate the intervention, spreading it across pupils 
and year groups as a whole-school improvement strategy. Inevitably, some schools have found it 
extremely difficult to achieve these levels of active involvement, either in the training or in the in-school 
and cross-school activities, and it seems that a more modest staff group form the core team in most 
both participating schools, though there is no doubt that those most actively involved are enthusiastic 
and committed advocates of the programme. 


Nevertheless, the message that CtG was an opportunity to trial ideas and approaches, with a view to 
wider adoption, has clearly taken hold, and we see ample evidence of this in the case studies, as one 
teacher remarked: 


‘Any changes we make we dont just focus ... on the fifteen, feedback affects everybody. It’s putting 
strategies in place that will help those free school meals children that will ultimately help everybody 
anyway...’. Another explained: ‘For example, in one of our CtG workshops, we did sessions where 
teachers had to bring in examples of how students respond to feedback. This has been a real success 
in our school and we’ve really driven it forward with staff, we've delivered a range of interactive sessions 
on marking and feedback during INSET days, regular senior and middle leader monitoring of marking 
and feedback, and we have a whole-school strategy whereby the teachers regularly give students time 
at the start of their lesson to read their comments and reflect on them.’ 


Indeed, in many of these schools we heard that the ideas for the interventions that were designed 
and developed for the pupil premium children had become interventions that were applied to 
and affected the whole class, and in some the impact from the programme was felt more widely 
across the year group. While this was clearly the major goal of the programme, it seemed that some 
participating schools were cascading practices almost immediately they were developed. An 
organisational culture that supports the adoption, adaption, and dissemination of new practices is thus 
a key determinant of the pace of development. 


The importance of appropriate leadership was also apparent in the responses to the survey: ‘There 
has to be ‘buy in’ at all levels, beginning with the headteacher. It needs a strong leader (who you can) 
access, and good relationships with all levels of leadership across the school for the impact of the 
project to be seen’ was one comment. 
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Target cohort 


However, putting the specific goals of the programme on one side, we will look briefly at what evidence 
we found that CtG led to interventions that reflected what has been put forward above as ‘effective 
practice’. Looking first at the issue of local context, we can see that there are some features common 
to most of the interventions that have been developed within the programme schools — such as the 
cohort size, the deliberate early targeting of children rather than looking for test year ‘cramming 
strategies’, the emphasis placed on the need to extend the social and cultural experiences of target 
pupils, and, perhaps most important of all, the integration of paraprofessionals as key deliverers of in- 
school support. However, there are also many differences, large and small, that underline what we 
might term ‘the present and the particular’ of specific contexts. 


Thus, for example we heard that in one school pupils were selected ‘... with the team leader for Year 8 
at the time, based on a combination of the data, paying particular attention to those not making expected 
progress in Maths and English. We had originally thought that would mean working with those with 
learning difficulties or challenges. Now, we find that it is also our able pupil premium students who are 
also vulnerable. Some of them have attendance issues, though nothing too chronic. Some of them have 
issues around parental engagement, or they have a lack of experience of Higher Education. They’re 
often the ones just not quite knowing what to do with their lives.’ 


In another school, the project leader ‘...spoke to teachers and looked at where the gaps were in our 
learning, and chose the target pupils based on that and on what we thought we could do for them. We 
also looked at the bigger picture; we didn’t kind of isolate it—we picked those children on whom we felt 
we could have a bigger impact straight away’. And in a third school the senior leaders ‘discussed every 
student to establish what interventions they were currently receiving, though SEN children were avoided 
because of the multiple interventions already in place... The 15 students finally selected represent a 
range in terms of ability, confidence, behaviour, and attitudes, so much so that beyond disadvantage, 
it is difficult to see any unifying characteristics’. 


In each of these examples we can see that the school made decisions based on local criteria and 
context — in one case the more able pupils are targeted, in another it is those the teachers feel can be 
most easily helped. In the third example the cohort is actually quite diverse, identified through a detailed 
scrutiny of a whole year group. 


Activities and experiences offered 


Just as the methods of identifying the target cohort were localized, so too were the support strategies, 
though typically it was teachers rather than school leaders who decided the details of these. Of course, 
there are common elements here too, and as noted above, many schools chose to create new social 
or cultural experiences for pupils in the target group. Nevertheless, a wide variety of such experiences 
are reported in the case studies, again, for the most part making use of local facilities and opportunities. 
Sometimes these were activities limited to individual schools, though most often it seems that schools 
planned trio activities together, even if the specific experiences offered to pupil groups were different, 
as they were, for example, in mixed phase trios. Sometimes the experiences planned were the same, 
though they did not take place at the same time, and in some instances it was possible to bring together 
pupil cohorts across the trio to share activities. It is clear from the comments of staff members who were 
involved that the school-to-school dimension was felt to enhance both staff and pupil 
experiences. ‘That’s what really struck me, the collaboration aspect of the programme and the 
opportunities for working together across boundaries — that’s what really, really made it.’ 


This view found strong support also in the survey, where respondents were very clear that the trios had 
added value to their own learning, much as they had to their pupils. They valued ‘... working with other 
professionals from different settings, being able to visit other schools and the CPD opportunities that 
came out of these’ and ‘collaboration with similar schools and with teachers facing the same 
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problems...’ Even when schools did not elect to do the same activities with their target pupils, they felt 
that the close relationship with trio partners was an important element of the programme. 
‘Sharing ideas was very helpful, and seeing that other schools have the same problems is reassuring. 
Actually we worked separately, but somehow having each other there was important.’ 


There is also evidence that the schools were keen to learn from best practice, and so appreciated the 
training workshops which were a ‘Good opportunity to share ideas and good practice’ and introduced 
them to recent research findings, encouraging them to adopt and adapt lessons from elsewhere; ‘the 
Challenge the Gap workshops were the starting point, and were intended to provide those attending 
with useful information (from research and reports) and stimulate ideas’; the reading materials that the 
participants were able access to dig around the topic gave them more to think about, and ‘this changes 
the way you look at things’. Workshops also enabled best practice to be picked up from other schools, 
as well as from inputs and resources provided by the programme. 


Finally, we see in the comments from participants that they understood this intervention was not a 
‘quick fix’, but rather was modelling a process that could and indeed must be repeated both 
within the school and with partner schools if the full impact was to be realised. ‘Over the two years, 
interventions have become more targeted and mapped. The interventions now include work on 
emotions and social skills. Prior to CtG all interventions were for academic subjects’, one participant 
reported. ‘As a result of the programme, whole school provision for PP pupils has changed. All PP 
pupils (now) have targeted intervention, they are named and noted on class action plans and 
Performance Related Pay is linked to the outcomes of PP pupils.’ A third participant described the 
impact of the programme within the school, and why the school would be adopting the ways of working: 
‘The impact of our work has been astounding. In the first year of activity, most children made two sub- 
levels of progress, and their wellbeing has improved beyond recognition. Some even visit our learning 
wall independently to get the resources they need to move on. Our pupil surveys show that confidence 
has significantly increased and that reading progress is much higher this year. We will definitely 
continue to work in this way in the future.’ 


Are there factors that act as barriers, limiting the impact of the programme? 


Inevitably, a number of limiting factors also emerged, though it should be emphasised that these were 
reported relatively infrequently. Only a small minority of staff in the case study schools and an even 
smaller minority among respondents to the questionnaire reported less satisfactory experiences, though 
these do need mention. For example, it is clear that while the trios worked well for schools from the 
same phase, in cross-phase trios the experience was rather more mixed. As one interviewee 
reported: 


‘It would have been far better if our trio had been a trio of primary schools ... because we were working 
on similar things, where it was difficult to find links between primary and secondary and the 
conversations that you were having when you were out on the training days ... you know, | just found 
myself having to go and join the primaries so that we could put into our context what the secondary 
school staff had been saying in their conversations.’ 


It is also the case that some schools withdrew, that is, did not complete the expected two-year cycle. 
Eleven of the initial pilot schools (n=39) opted not to carry on, though the reasons varied. As the 
programme moved on, the dropout rate increased: of the 139 schools that started year two of the pilot, 
only 58 continued into a second year of the programme. As noted above, trios embracing both primary 
and secondary schools seemed to offer less to primary partners. One school seemed to be 
overwhelmed by wider performance problems and could not really cope with a further intervention 
programme, suggesting that the timing of such initiatives needs to be planned in light of the 
competing priorities a school may be facing. Some schools complained about the time out of school, 
the distance needed to travel to workshops, or questioned whether separate workshops for the different 
groups were really necessary. When budgets were tight, it seemed that some schools found it 
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difficult to resource the school-based and cross-school activities that add value to the 
programme, however worthwhile they felt these were. This was particularly the case in small schools, 
where cover is not readily available from colleagues and almost all staff develooment during the school 
day needs to be supported by bought-in cover. In some cases, schools had difficulty recruiting ‘new’ 
partners, so the transition from Accelerator to Lead school could not take place as planned. 


As noted above, some schools that started the pilot year did not continue into year two, but those that 
did were joined by a tranche of newly recruited schools in the 2013/14 school year. In all, 134 schools 
signed up for the first year of the extension phase, though 12 of these withdrew before completing the 
year. A further 54 schools opted not to continue after one year of the programme, leaving 58 schools — 
not much more than half- that continued into the expected second year of participation. These schools 
were joined by another 53 new recruits, meaning the overall cohort was roughly the same size at the 
beginning of both Year 1 and Year 2 of the extension, but similar total participant numbers mask the 
level of turnover taking place among programme schools. We do not have data regarding the 
dropout/continuation rates between Years 3 and 4. 


The data we have suggests a mix of reasons for withdrawal, similar to those offered during the pilot 
phase, plus the additional burden for Accelerator schools of paying the programme fee that had 
been introduced. It should be noted however that during the pilot phase, CtG had taken steps to ensure 
that many of the problems mentioned would no longer arise, with the introduction, inter alia, of twilight 
workshop sessions, local hubs, and within-phase school trios. 


The availability of resources within the programme schools to fund joint activities between staff groups 
has also been mentioned as a key factor. Time was mentioned so often as a facilitating resource 
that it seems likely impact was limited where sufficient time could not be made available, as was 
sometimes the case in small schools, or when other priorities, for example an Ofsted inspection, 
suddenly arose. What is also important here is the additional cost of releasing staff to visit each 
other’s schools, to observe one another’s practices and work together on joint practice 
development. This is a relatively expensive activity, especially for the small primary schools that were 
recruited after the initial pilot year, wnere many such activities are simply not feasible without paid cover. 
Where such release cannot be funded, the spread of ideas moving between programme schools — 
perhaps the most important resource of all — is significantly curtailed. 


Can these barriers be overcome? 


Participants made a number of suggestions that might overcome the problems identified. There was a 
feeling that the amount and use of time spent in workshops could be improved; for example, 
more could have been focused on the sharing of experiences from the wider network. One respondent 
to the questionnaire remarked: ‘We spent too much time filling in forms about what we had done. | 
would have liked to hear more about what other schools were doing’, while another felt that the balance 
between ‘workshop’ time and time available to meet as trio partners could be improved: ‘/ think that we 
could meet more often as trios... not as a local group so often ... practical examples (from other schools) 
are more useful than lectures....’ 


Other participants continued to question whether separate workshops for different staff groups 
were necessary, pointing out that it was actually quite difficult for the in-school groups to find time to 
meet within school. Some felt that the workshops offered ‘quality time’, that school teams might better 
use to plan and discuss the progress of their own school programme, though often these same 
individuals were very positive about trio meetings and activities. Time spent in school teams, or in 
activities with colleagues from trio schools, was generally seen as more productive than ‘sharing’ with 
the wider cluster. 


Some of the post pilot schools — who indicated that they would not be signing up for a second year — 
cited the cost of the programme as prohibitive, particularly for small schools. They also 
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questioned whether a second year of the cycle would actually add much value, pointing out that once 
they have been initiated into the programme, enjoyed access to the materials and experienced 
working with the trio partners, they felt that they could replicate this process themselves, without 
the need for central facilitation and workshops. They also wondered whether continuing to strengthen 
relationships and collaboration within the initial trio offered a better platform for continuing 
professional development than recruiting new partners and having to once more build the 
relationships and trust that underpin authentic collaboration. 


Finally, it should be remembered that existing attitudes can be an important barrier to 
improvement. One of the schools responding to the questionnaire was clearly somewhat disappointed 
with their experience, stating (in response to almost every question) that the school already did all that 
was possible to ensure the progress of disadvantaged pupils, and had found nothing ‘new’ in the 
programme. Of course, the majority of schools feel that they do all they can, but despite this remain 
open to new approaches, even new approaches that draw on familiar elements. There would seem to 
be little point in joining a programme if you have already decided that your own practice is already 
ahead of what is being proposed, so presumably this school found that initial expectations were not 
met. Nevertheless, the disposition to believe that we are essentially hard-working people doing all we 
can has been identified as a key factor (Janis, 1982) in poor decision-making, and a barrier to 
development. Perhaps more active leadership is sometimes required, to ensure that the staff 
members participating in the programme remain open to the ideas and approaches from 
outside. 
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Conclusion 


Key conclusions 


1. The project found no evidence that Challenge the Gap (CtG) increased average attainment for 
either primary or secondary school pupils, overall. The security of the primary school results is low 
to moderate, and the security for the secondary school results is low. 


2. The findings are different for children eligible for free school meals. FSM-eligible children in CtG 
primary schools made 2 months’ additional progress compared to similar children in other schools, 
while FSM-eligible children in CtG secondary schools made 2 months’ fewer progress compared 


to similar children in other schools. The smaller number of FSM-eligible students in the trial means 
that these results are less secure than the overall findings. 


3. Teachers and non-teaching staff from participating schools were extremely positive about the 
involvement of their schools in the CtG programme and valued both the professional development 
opportunities it provided and the opportunity to collaborate with partner schools. 


4. CtG is a flexible programme that allows lead schools to share best practice. There was large 
variation between the strategies that were adopted by the schools in the programme. 


Limitations 


As with any research, this evaluation was subject to a number of limitations. First, due to the restrictions 
of school recruitment and subsequent attrition, the study was not fully powered to detect small changes 
in attainment (especially in relation to secondary schools). It is possible that small effects of CtG are 
undetected by the evaluation. Second, as randomisation was not possible, it is more difficult to attribute 
any impact in pupils’ results directly to CtG. This is particularly pertinent as, third, the CtG programme 
serves as a framework for a multitude of practices and innovations to operate. It has not been possible 
to capture this variation in the quantitative analysis, meaning that some elements may have proved 
successful, but have not been identified as part of the assessment of outcomes. 


Interpretation 


Results indicate that there may be small gains for FSM eligible pupils in primary schools. Although 
results are not seen to improve for the wider cohort, this does address the specific challenge of 
addressing the attainment gap. However, these gains are seen to be very marginal. For secondary 
schools, results indicate no change, though in the case of Lead schools, there is some evidence to 
indicate that the wider cohort benefited. 


In considering the generalisability of these results, it is worth considering the make-up of the research 
sample. The evaluation reflects schools with a higher than average FSM eligibility and gap in 
attainment. It is not clear to what extent schools with a closer-to-average gap might choose to engage 
with CtG or what the outcomes might be. It is also important to note that many schools did not continue 
their involvement with CtG. 


It is also worth considering many felt that participation in CtG led to an improvement of variables 
associated with improved academic performance such as attitudes to school, and improved attendance. 
Additionally, participants reported many other, wider benefits, including enriched pupil experiences, 
enhanced pupil self-esteem, high quality professional development opportunities, increased awareness 
of possible strategies and approaches, raised expectations of staff and pupils alike, better motivated 
and more purposefully used teaching assistants, and improved leadership and teamwork within the 
school — which they feel have derived from participation in the programme. However, it should also be 
noted that several factors were associated with effective implementation, including appropriate senior 
leadership support and protected time to prepare for and reflect on the intervention. 
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Appendix 1: Survey of Challenge the Gap Programme 
Schools 


In the autumn of 2015 all CtG schools were invited to participate in an online survey that sought to 
collect their experiences in the programme. The development of the survey questions was informed by 
the data collected from case-study schools and from a number of workshops and evaluation sessions 
offered to CtG schools that evaluation team members attended. The survey was publicised via the 
Challenge the Gap programme team, and several ‘reminders’ were issued to encourage school to share 
their views and experiences. Despite these efforts, the response from schools was somewhat 
disappointing. 


30 responses were received, though scrutiny of IP addresses suggested that these responses likely 
came from a sample of 27 different schools, as some addresses were duplicated. However, the 
response rates needs to be seen in the current context in which CtG schools operate. As noted 
elsewhere, though the programme was largely funded through a grant from EEF during the first year of 
operation, it has since developed as a commercial venture, with schools paying what they see as a 
substantial fee for their participation. In such circumstances, it is clear that the schools have thought 
about participation carefully before joining the programme, and have already themselves concluded 
that the costs and potential benefits represent a sensible investment. Further, as the programme is 
essentially a ‘product’ which they have already purchased, whatever their experiences there seems to 
be less interest in external evaluation, with attention heavily focused on what impact there has been 
within their own schools. It is, therefore, understandable that the response rate was not as high as we 
might have hoped. 


The answers that were submitted revealed clear patterns about both the strengths of the programme 
and about those aspects which were more difficult to deal with in the schools. Overall, the responses 
were overwhelmingly positive, with only a small minority of schools that had found their involvement in 
the programme disappointing. 


Below, the results of the survey responses are briefly summarised, question-by-question. Where 
appropriate, questions are introduced by a bar chart showing the distribution of scaled responses, 
followed by a sample of the texts gathered from the associated comment boxes. Finally, a paragraph 
summarising these responses is offered. 
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Q1. Do you feel that Challenge the Gap achieved the objectives set for it in your 
school? 


There has been 
substantial... 


There has been 
some impact 


There has been 
little impact 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


e ‘Early to say, has had good outcomes for students and staff involved but don't know about attainment 

data yet’ 

‘Hard to measure real progress at this stage, but both staff and students very positive’ 

‘Pupils have really benefited, good PD for staff as well’ 

‘It has helped us focus and also to confirm our thinking on what works best for disadvantaged pupils’ 

‘It has highlighted the need to target all underachieving children. More staff members are alert to the 

barriers to learning these children face. We have concentrated on children who do not have a positive 

home life. A lot more work is now being done around social skills and understanding emotions’. 

e ‘Our PP learners are far more high profile and the gaps between PP and non-PP pupils are the 
narrowest they have ever been’. 

e ‘think we can see that things we did with the children we have concentrated on are extending across 
the year group, reaching more children’ 

e = ‘The project definitely helped improve the attitudes of some learners...’ 

e ‘The collaboration element was very powerful’ 

e = ‘The profile of Pupil Premium children has been raised significantly and the gap between PP and non- 
PP pupils has never been so small at our school.’ 

e ‘Focusing on a target cohort helped us to think about the many children in the school facing difficult 
backgrounds. | think the impact went well beyond the target pupils’ 

e ‘The children have progressed quite well. It has become a central part of the Teaching and Learning 
agenda and has a Clear place in the SIP’ 

e = ‘Due to the high ratio of PP within our school, a lot of the practice we did was what we did anyway for all 
of our children. Comparison data between the CtG cohort and the Non CtG showed minimal difference’ 

e ‘Initiatives did not accelerate learning or sufficiently close the gap between PP and non PP.’ 


Summary 


The respondents, though largely very positive about the impact of the programme, are evenly split 
between those who feel they have already seen accelerated progress made by targets pupils and those 
who are seeing other benefits that they believe will lead to improved progress in due course. Such 
benefits include increased awareness of the who the PP premium pupils in the school are, and also of 
the challenges they face, improved attitudes among pupils and expectations among staff, and high 
quality professional development experiences. Several references were made to the fact that strategies 
and practices developed for the target cohort were being spread more widely within the school. 
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However, two schools were less enthusiastic. In one, there was clearly a feeling that the programme 
offered nothing ‘new’- indeed the comment from this school that most of the activity ‘was what we did 
anyway for all of our children’ was repeated in every comment box. One other school reported that there 
had been little impact on ‘the Gap’. 


Q2. Based on your experience in the programme, what would say are the most 
important features of the CtG approach? 


e = ‘Involving the paras directly, and working with colleagues from trio schools’ 

e ‘Without doubt he trio- they have been great! In school, making everybody aware which students we are 
trying to help has been key 

e ‘Definitely working with trio schools/colleagues- really good to share ideas and strategies’ 

e ‘School-to-school collaboration, networking -especially cross phase, upskilling leaders through their 
participation in the programme’ 

e = ‘Learning about barriers to learning and successful strategies based on IEP. The involvement of staff at 
different levels (teachers, leaders and paras). Emphasis on themes of academic competence and 
resilience’ 

e ‘Recognising which children need regular support. Availability of funds (PP) to support school based 
action. Working with other schools’ 

e = ‘It really exemplifies distributed leadership; it allows cross subject collaboration to take place. It 
empowers para- professionals and also creates JPD opportunities working across schools.’ 

e = ‘Working with other professionals from different settings, being able to visit other schools and the CPD 
opportunities that came out of these.’ 

e = ‘Raising the aspirations of the most vulnerable -JPD with other colleagues -Developing resilience and 

academic literacy levels in pupils’ 

CPD, Academic literacy, Big idea of Maths, getting everyone involved’ 

‘Sharing good practice across and within schools’ 

Sharing and networking between schools, time to do this’ 

UPD is very important, having opportunities to share good practice. Raising the aspirations of the most 

vulnerable learners has been a key.’ 

e = ‘Targeting the family, not just the pupil. Being creative and also practical about the support you provide.’ 

e ‘Collaboration with similar schools and with teachers facing the same problems. Clearer objectives for 
staff in school-especially teachers and classroom assistants working together’ 


Summary 


The responses here, apart from references to time and resources, seem to cluster around three key 
factors: the benefits of working in a trio of schools, the impact of the programme on the 
paraprofessionals involved, and the importance of knowing exactly who the PP pupils are. There are 
many references to how enriching teachers have found their collaboration with colleagues from other 
schools in the trio, visiting other schools, and participating in joint practice development. This is the 
most striking finding. There are also several references to the way teaching assistants or 
paraprofessionals involved directly in the programme have contributed to its success, and have, 
themselves, grown through the experience. The targeting of specific pupils (and, in some cases, their 
families) is another important factor, and it is this that enables a relentless focus on numeracy and/or 
literacy with these pupils to be developed across the curriculum. 


Q3. Based on your experience, what do you feel are the key factors in successful 
implementation of the programme within school? 


e = ‘Time to plan, funds for meetings and activities’ 

e ‘Time, more time! Good paras who are willing to accept the challenge themselves’ 

e = ‘Making everyone aware of which children are the PP children in the school. Time to plan things 
properly. A budget for additional activities. Good paras!’ 

e ‘Good leadership. Successful facilitation. Good admin support. All those involved ‘buying into’ the 
programme.’ 

e ‘Grounded in research about what works- from evidence from EEF. Getting staff to think about their 
own pedagogy, and embedding the interventions into SEF Cascade model’ 
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e =: ‘Staff working together purposefully. Time to plan. Regular reviews of progress’ 

e = ‘There has to be ‘buy in’ at all levels, beginning with the Headteacher. It needs a strong leader with 
access, and good relationships with all levels of leadership across the school for the impact of the 
project to be seen.’ 

e ‘Consistent staffing, regular meetings and high levels of commitment from individual staff members 
involved’ 

e ‘Whole school buy-in is essential. The whole school need to be aware of who the most vulnerable 
learners are and all must be committed to raising the profile and accelerating the progress of all PP 
individuals, beyond those pupils in the Challenge The Gap cohort’ 

e ‘Effective planning and collaboration within the team, which means having time to meet on a weekly 
basis...’ 

e ‘Ensuring enough time for discussion, planning and implementation of interventions is allocated and 
protected’ 

e ‘Assess the needs of your unique cohort irrespective of who is in your trio’ 

e ‘Whole school buy-in to the programme is vital. Every adult in the school knows who the Challenge the 
Gap pupils are, and all teachers and support staff are aware of the PP children in their classes and 
actively aim to support and accelerate their progress.’ 

e ‘Time was a big factor, the head gave us time to meet and plan Knowing which students are in the target 
group before you arrange timetable the response of the students keeps you going!’ 

e = ‘Due to the high ratio of PP within our school, a lot of the practice we did was what we did anyway for all 
of our children. Comparison data between the CtG cohort and the Non CtG showed minimal difference.’ 


Summary 


In terms of what facilitates effective implementation, resources are clearly seen as key. Time seems to 
be the most important resource, closely followed by the availability of staff ready to commit to the 
programme at three levels- leaders, teachers and support staff. Equally important- if less tangible- 
resources include ideas, about what can be done within the programme, enthusiasm, and effective 
planning- respondents speak of the need to organise and coordinate activities and to work purposefully 
together. PP funding has also played a role, though this receives less mention than might have been 
expected. 


However, the benefit of ensuring as many staff as possible know which pupils are being targeted is 
again a theme that emerges, perhaps implying that teachers have experience of interventions where 
this has not been the case. This reinforces the message coming through the comments that effective 
communication is a major factor in determining whether school level interventions are successful, and 
this is perhaps more difficult to achieve in schools than we might imagine. 


Q4. Do you feel there are any school level factors that make the implementation 
of the CtG Programme difficult? 


e 6: ‘The timetable- students are in different groups most of time. And it is hard to find time to get together 

with colleagues to plan and evaluate’ 

‘Timetable does not bring whole group together. Release for cross school visits hard to arrange’ 

‘Time to plan things! Hard to fund the programme in a small Primary school’ 

‘Schools paying for it and not seeing quick wins...’ 

‘Time to plan and meet with colleagues involved the programme’ 

Finding time to plan! 

‘The time needed for meeting staff is difficult- this is a big factor for trio teams e.g. para professionals 

who have contracted hours. Also difficult for trio teams as there are many meetings after school. The 

project is also requires quite a big effort from trio leads as they have to arrange meetings with their team 

and attend meetings with the CtG lead. In addition, the impact of any of the trio projects depends on the 

drive of the lead and team members (as with anything). Plus, the trio members can feel disconnected 

with the programme if the training they receive from trio lead is not clear.’ 

e ‘Some schools did not attend workshops consistently or send the same staff, | think consistency is 
important. 

e =. ‘Releasing staff at the same time can be difficult and meeting up across the schools for JPD is very 
difficult due to the need for classroom cover.’ 

e ‘Parental engagement is important but difficult to engage’ 

e ‘Monies to cover supply, and the extra staffing needed.’ 
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e Engagement of class teachers not consistent- they have many priorities’ 

e © ‘Time is an issue - trying to ensure that JPD happens as partner schools cannot always find time to be 
released.’ 

e ‘Despite being given some time, we could have done with more time in school, after school meetings are 
less effective. | would also have liked to be able to visit the trio schools more often.’ 


Summary 


To a large extent, the absence of the factors identified above- particularly time and clear goals and 
plans- were identified as posing most problems. When there was insufficient time, this was most often 
attributed to the shortage of financial resources, funds necessary to buy supply cover to release 
teachers, particularly at the same time, so planning could be a collaborative process and be carried out 
in the school day. There were a number of references to the additional pressures created by after school 
meetings, and to the other priorities that teachers must attend to. A further problem was the ways target 
pupil groups were identified or distributed across the year group. In many cases these pupils were 
drawn from different forms, or taught in different sets, so that it was hard to get them together as a 
group within the timetable. And, while it had been recognised that parental engagement was important 
to pupil progress, this was not easy to secure. 


Q5. In your view, what impact has the Programme had on the attitudes of the 
teachers involved? 


There has been 
a substantia... 


There has been 
some impact 


There has been 
little impact 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


e ‘They are more motivated and work more closely with paraprofessionals’ 

e ‘Early days but impact has started in knowledge sharing on research in PP, the teachers are 
enthusiastic, they are really up for it’ 

° ‘We are all committed to the programme. We have seen that we can achieve improved outcomes if we 
have the necessary resources and commitment’ 

e ‘CtG has given trio leads the power and knowledge to carry out leadership. It has given them the ability 
to network with teachers from other schools and this is also very important’ 

e §=©‘All in my setting, including myself have increased in confidence and several gained promotions....’ 

e ‘The teachers who have been involved have noted that the programme has really impacted on their 
practice. They are more aware of the needs of vulnerable and disadvantaged pupils and their new 
learning is being disseminated across the school.’ 

e ‘Lessons have been learned, resources have been discovered, and new strategies have been 
implemented’ 

e = ‘All teachers involved have felt they have benefited greatly. The year 6 teacher involved last year 
impacted massively on her cohort - the gap closed entirely at L4 in Maths and PP pupils outperformed 
others in reading.’ 
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° ‘With such high PP numbers, these children are in the forefront of the teachers minds, and it was difficult 
to justify providing this support from some children and not others. Also a lot of the strategies were 
already embedded in school practice.’ 


Summary 


Almost all schools responding commented that the programme had generated increased enthusiasm 
and commitment among the teachers involved; in the majority of cases substantial changes on attitude 
were reported. Much of this related to greater clarity about purposes (who are the children whose 
progress we are seeking to accelerate?) and to the sharing of practice, both within and across schools. 
Some noted that improvements in pupil outcomes can be accompanied by improvements in teachers 
own prospects, through increased responsibilities, changes in role and even promotions. There was 
very little dissent from this view, though one school remained adamant that little had arisen from 
participation in the programme that was new, and questioned whether the additional resources 
channelled towards a small cohort of pupils was an appropriate strategy in schools where PP pupils 
were in a majority. 


Q6. In your view, what impact has the Programme had on the classroom practice 
of teachers involved? 


There has been 
substantial... 


There has been 
some impact 


There has been 
little impact 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


e =‘ think it has probably changed the way they think about some pupils, | know it has for me, so they are 

more supportive to those pupils. At the same time, | think it has raised expectations. 

‘| have certainly changed the way | work with target students, and with the TA that supports them..’ 

‘More focused on progress of these students...’ 

‘Actions targeted more clearly. Cooperation with teaching assistants improved’ 

‘.once the project and the work became whole-school, with the acceptance from headteacher, teachers 

took on the work and we are seeing it develop’ 

e = ‘more willing to trial new ideas and also consider the emotional strain children are under in a classroom’ 

e ‘The gaps between PP and non-PP pupils last year in the cohort of the teacher involved in the 
programme disappeared in Maths at L4 and PP outperformed non-PP in reading in L4, so they must be 
doing something different..’ 

° “..we’ve seen the introduction of new interventions e.g. Talk Boost; focus on resilience and 
perseverance, and some strategies to support meta-cognition’ 

e ‘The teachers involved are using strategies to develop academic literacy in classes and are developing 
resilience in the learners within their classes. This is being cascaded cross-phase by the teachers’ 

e =| think it has probably changed the way they think about students, | know it has for me, so they are more 
supportive to some students. At the same time | think it has raised expectations. 


‘Hard to say- can you really measure these things? | think we already do everything we can...’ 
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Summary 


While schools reported substantial changes in teacher attitudes, there was less confidence that these 
had translated into changes in the classroom practices of the teachers involved. Of course, attitudes 
are often seen as a leading variable, which shift in advance of behaviour (Likert 1976). Others argue 
that while subsequent behaviour changes generally tend to be in the same direction, they are less 
marked, so actual behaviour changes rarely reflect the scale of attitude change. Either way, many 
schools clearly felt that that they were in the time-lag between attitudinal and behavioural change, 
though, and were able to point to specific developments taking place, and felt optimistic about these. 


There had clearly been a series of interventions stimulated by the programme, largely focused on the 
PP target cohort, though in some cases spread more generally across classes or year groups. However, 
it was also clear that many schools felt they were at the beginning of a process that would take some 
time to realise its full benefits. 


Q7. In your view, what impact has the Programme had on the attitudes of the 
classroom assistants/paras involved? 


There has been 
a substantia... 


There has been 
some impact 


There has been 
little impact 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


e ‘Fantastic response. The two who are involved have been a real help, and have great relationships with 
the students’ 

e =. ‘They have been really helpful, and | was surprised by how much they know’ 

e ‘think paras gained most among the staff- they love it!’ 

e ‘Comments to date show a real impact and appreciation of their personal learning and being involved in 
a whole school project’ 

e ‘There has been a marked increase in interest in children's progress They are much more confident. 
They have been really helpful, and | have been surprised by how much they know’ 

e ‘| believe the paras in our school have felt privileged and empowered at being picked to be part of the 
team. They were very focused and have grown through the experience’ 

e ‘have gained considerable amounts of confidence, | have been more willing to share new ideas with 
other staff and taken on greater levels of responsibility’ 

e = ‘The TAs involved are promoting independence and resilience in the learners, rather than focusing on 
task completion, which did happen in some cases’. 

° ‘The TAs involved have enjoyed and benefited greatly from their involvement. They have led carousels 
and this has allowed them opportunities to develop many skills such as public speaking. The TAs 
understand how to best support PP pupils in all classes and they have developed strong relationships 
with the pupils in the groups’ 

e The TAs were very negative about the experience. They felt that in order to feel part of the process, their 
meetings should have been held with the school leads present. 
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Summary 


There was even greater consensus that the programme had a positive influence on the attitudes of the 
teaching assistants/paraprofessionals who had been involved. Many felt that the impact had been life- 
changing, for the first time integrating these staff members fully into the broader teaching and learning 
processes, and there had accordingly been corresponding changes in attitudes to both the work and 
the pupils they work with. Many comments mentioned a closeness that had developed between these 
staff members and their pupils, indicating improved relationships and more effective support. 


The comments (largely written by teachers?) single out the ‘paras’ as the group whose attitudes and 
expectations have changed most. However, though acknowledged less frequently, it is clear that many 
of the teachers making these comments have themselves experienced significant changes in their own 
views regarding the role and potential of support staff. There seems to be increased recognition that 
pupil outcomes can be directly influenced by the work of support staff, and a wider appreciation of what 
support staff do and can contribute to the acceleration of pupils’ learning. 


Q8. In your view, what impact has the Programme had on the classroom roles 
of the teaching assistants involved? 


There has been 
substantial... 


There has been 
some impact 


There has been 
little impact 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


e | think they are working closer with teachers and with pupils’ 

e ‘They are much more interested in lesson plans and objectives. they have their own goals, and set goals 
for the students’ 

e = ‘Like I said, they gained most, and | think teachers appreciate them more They took ownership of the 
project’ 

e ‘We are lucky to have a couple of graduates working as TAs, this has really given them a chance to 
become involved, and their response has been amazing’ 

e = ‘The joint planning with the para has made a big difference, and in class | know that she is more aware 
of what the lesson goals are, and | can trust her more to work on these with the pupils’ 

e ‘started working on CTG as a level 2 TA and | am now employed as a HLTA. This has allowed our 
school to significantly reduce the number of supply teachers used, something that unsettled our 
children, knocking onto behaviour and attainment.’ 

e ‘The TAs roles have changed and TA training has taken place since the start of the programme, 
focusing on developing resilient and independent learners.’ 

e ‘Teaching assistant led groups, planned delivered and taught in a practical way’ 

e = ‘The paras involved are promoting independence in the learners rather than focusing on task completion 
which was initially an issue within the school.’ 

e ‘1 think it depends on the teacher, some have been more prepared to work as team than others’ 

e Depends how they are used, more involvement in some classes but not all, you can’t make people 
cooperate’ 
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Summary 


There are similar perceptions about the impact of the programme on the working practices of the 
support staff involved. Though, as with teachers, behaviour change is seen as less marked than 
attitudinal change, there is a smaller difference between these variables, and many specific examples 
of things that are being done differently were reported. It is reported that support staff are both taking 
more responsibility and showing greater initiative in the classroom, and are also providing additional 
support outside the classroom too. 


There are one or two less positive comments, which seem to centre on relationships. Obviously, the 
relationship between teacher and support staff member needs to be strong if a full contribution of 
support staff in the learning process is to be achieved. Sadly, there seem to be a few instances where 
the relationship is not strong enough to allow this to develop. 


Q9. In your view, what impact has the Programme had on the attitudes of the 
pupils involved? 


There has been 
substantial... 


There has been 
some impact 


There has been 
little impact 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


e ‘Self-esteem and confidence levels up. Better attitude to school. Mostly improved behaviour..’ 

e ‘Attendance and behaviour both improved. More confident. Progress OK- hard to know till they face real 
test’ 

e Evidence of some impact but early days as new cohort, have evidence of improvement recorded in 
individual cases by teachers’ 

e ‘More confident. Happier in school. feel special... more resilience in learners’ 

e 6*...over the two years over 30 children gained positive experiences and staff were able to build lasting 
relationships with these children, most of whom are vulnerable in one way or another- young carers or 
significant SEBD...’ 

e ‘The pupils who have been involved in the programme are so positive about school and their learning. 
Their aspirations have raised and they are used to speaking to a range of different audiences. Previous 
pupils have adopted a mentoring role towards other CTG members and are keen to talk about how the 
programme has positively impacted on their lives’ 

e *.... ‘The children definitely gained from interventions such as lunch time clubs... They have grown in 
confidence, and | worry what we will do next year to sustain them!’ 

e = ‘The pupils involved have blossomed! They are full of enthusiasm, resilience and pride to be involved in 
the project. Past pupils are positive role models and mentor others’ 

e ‘They have grown so much in confidence- | worry what we will do next year to sustain them! 

e ‘The children did enjoy some of the new opportunities that arose from being a part of CtG, but this did 
not impact on overall attitudes and learning’ 

e ‘Bit mixed- most of them seem to be enjoying the attention and the activities, but a couple are harder to 
pullin...’ 
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Summary 


Responses are evenly split between ‘substantial impact’ and ‘some impact’ on pupil attitudes, and in 
most cases it seems changes can be readily identified. The main areas of impact seem to centre on the 
attitudes to school and schooling, and attitudes to self. Many schools report improvements in pupils’ 
attitudes to school, and comment on new levels of enthusiasm and interest. Others report that pupils 
seem more confident, more at ease, and show increases in self-esteem. 


In a couple of cases, comments question whether these apparent changes are likely to lead to useful 
behaviour change- or whether the activities and experiences provided to these pupils are just adding 
some fun into the school diet. Of course, the pupils enjoy such activities, but will this impact on their 
wider attitudes towards school? 


Q10. In your view, what impact has the Programme had on the behaviour of the 
pupils involved? 


There has been 
substantial... 


There has been 
some impact 


There has been 
little impact 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


e ‘Attendance and attention generally have improved. Sometimes, they have been really excited- like the 
various trips. It is the first time we have seen them excited about school!’ 

e ‘Most are behaving better, less cases of low level disruption Some parents more interested in their 
children's progress’ 

e ‘Better motivation. Less disruption and some have improved attendance - not all.’ 

e ‘The children are beginning to make the connection between behaviour, attitude and attainment, that 
they are responsible for their own actions, staff are there to help, not do it for them.’ 

e ‘A positive ‘can do’ attitude’ 

e ‘Most of these children are getting more attention than they did, and when they get used to it of course it 
starts to change the way they think about school. But in some cases it is beginning to change the way 
they think about themselves, and that is the real benefit.’ 

e ‘Attendance and attention improved Sometimes they have been really excited- like the trips. It is the first 
time we have seen them excited about school’ 

e = *...mixed again, but | think they are mostly better motivated in school’ 

e ‘Some of the children's behaviour is still undesirable at times but the majority of the children are positive 
role models in the school now’ 

e = ‘Behaviour was not actually a concern with these children’ 


Summary 


Responses here are rather more cautious, with only a minority of schools reporting substantial changes 
in the behaviour of target pupils. However, where they have been recorded- changes in attendance and 
improved behaviour, for example, these do seem significant outcomes. The majority of schools see less 
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dramatic behaviour changes, though increased levels of excitement and enjoyment are reported more 
generally, and these are clearly important for motivation levels and are likely to encourage resilience, 
which has been identified as a key influence on the attainment levels of these pupils. However, a small 
number of respondents remain more sceptical about the impact any of this will have on actual 
attainment. 


Q11. In your view, what impact has the Programme had on the progress made 
by the pupils involved? 


There has been 
substantial... 


There has been 
some impact 


There has been 
little impact 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


e ‘Pupil progress has been good but the greatest impact can be seen in the soft data, where the attitudes 
to learning and school have improved notably.’ 

e = ‘Pupil progress has been good; attainment may be below expectations but the progress made by most is 
accelerated.’ 

e ‘think they have made gains, though we won't really know if this can be sustained for some time yet’ 

e ‘Not sure yet. There is definitely some improvement, but building literacy is a slow business, and we 
need to keep focus on this over more than one year’ 

e ‘Progress accelerated, but there is a long way to go. | don't think you can expect the gap to close in one 

year’ 

‘There has been some improvement in Maths for the PPG pupils who participated in the programme...’ 

‘Too early to say, but attitudes to school are generally better, so this is promising..’ 

‘Hard to say yet, but the other signs are good, and we have high expectations’ 

‘Focused interventions have had some impact, but whether it justifies the effort is hard to estimate at this 

stage’ 

e Changes in assessment systems have made it really difficult for us to track the pupils this year. We think 
that they are progressing, but at the same time we are not absolutely confident that we have got the 
levelling right yet’ 

e = ‘Many of our pupil premium children are more able and the programme was really geared to less able 
children’ 

e ‘With such high PP numbers, as noted above a lot of the strategies were already embedded in school 
practice. It is hard to say that any improvement is because of this one intervention.’ 


Summary 


Probably this is the most important question- did the participating schools feel that the programme was 
indeed accelerating the progress of target pupils? The responses are encouraging, though not 
overwhelmingly so. Only two schools reported substantial increases in progress. The majority were 
more cautious, indicating that there had been some progress and displaying optimism, though often 
pointing out that it would be some time before an evidenced judgement could be made. However, it 
could be said that expectations among teachers remain high, and the early results secured have, for 
the most part, not disappointed. 
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Some mention was made of the changes to the assessment regime in Primary schools, which teachers 
felt had made the tracking of pupil progress more difficult over the past year. There were three schools 
that reported little impact from the programme. 


Q12. In your view, what impact has the Programme had on the school’s wider 
intervention strategies? 


There has been 
substantial... 


There has been 
some impact 


There has been 
little impact 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


e ‘We are planning differently for next year, need to keep this going with the students and to spread the 
approach to the wider group’ 

e ‘It has made us think about whether we were doing all we can for these children. Just putting up their 
photos in staff room had an impact on teachers’ 

e ‘Early days but cascade model working especially where HTs and the SLT are actively involved in 
planning the intervention programmes and they have been embedded into the SEF’ 

e = ‘All teachers now have to plan intervention for PP pupils. PP pupils progress is discussed at termly 
Teaching and Learning Review Meetings and performance related pay is linked to the progress of PP 
pupils.’ 

e ‘Over the two years, interventions have become more targeted and mapped. The interventions now 
include work on emotions and social skills, prior to CTG all interventions were for academic subjects’ 

e §©‘As a result of the programme, whole school provision for PP pupils has changed. All PP pupils have 
targeted intervention, they are named and noted on class action plans and Performance Related Pay is 
linked to the outcomes of PP pupils.’ 

e ‘Incorporating some of the CPD and keeping logs to reflect on learning’ 

° ‘The school has benefited from learning from other schools and introduced Forest Schools, Talk Boost, 
and a focus on resilience. We have had some discussions about whether we need to do more in year 7. 
Students we know are struggling fall further behind in year 7, and we are thinking of putting more 
resources and attention on them at this point. But it is hard, because everyone is worried about KS4 
results.’ 

e ‘We can see it has benefited the target group. But was it best way to spend money? SMT review thought 
it had been worthwhile, and that we had learned from project, but nevertheless they did not want to 
continue it....’ 


Summary 


It is clear that involvement in CtG has influenced programme schools’ wider thinking about intervention 
strategies and targets. One outcome has been to place more attention on PP pupils generally, not just 
members of the target cohorts, and many schools have reported the adoption of systems to ensure that 
all teachers are aware of who these children are, and have clear strategies to support them in class in 
future. At the same time, information on which pupils are in the target cohorts has been disseminated 
much more widely among staff than is general with targeted interventions, and it is clear that schools 
feel that this in itself creates momentum around the efforts to accelerate progress. 
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A second outcome has been the spreading of intervention strategies between schools- the trios, and 
relationships between the teachers within the trio schools have been key in this, and there are many 
examples of schools ‘borrowing’ approaches from one another. There is also some evidence that 
collaboration with other schools encourages experimentation, so that teachers are more willing to try 
new approaches and worry less about getting it wrong, though the pressures from external testing are 
still a major determinant of the schools’ intervention priorities. 


Q13. In terms of impact, how useful/important do you think the local workshops 
were? 


Very useful 


They had some 
influence 


Not very useful 
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e ‘Some were good, but they seem drawn out a bit. Why can't we do them together as a school all the 
time? Why do we have to send so many reports in- who looks at them?’ 

e ‘They were most helpful at the beginning- as we got into things they were mainly just meetings to talk 

with trio partners but still that was valuable’ 

‘Liked the opportunity to talk to colleagues from other schools, hated filling in the forms!’ 

‘Real quality in delivery and information. Really insightful.’ 

‘Liked meeting other teachers- gives you confidence that everyone has same problems’ 

‘They were a chance to talk with other staff and although most did not include much new learning for 

myself, they did give one a chance to just think about the way we work. A rare opportunity during the 

working week’ 

e ‘The local workshops have been very useful as they have allowed all staff to develop in their roles, to 
develop better practice based on the knowledge and expertise of others and it has been particularly 
powertul releasing the TAs, teachers and leads at the same time, we don’t get much chance to do this in 
school’ 

e = ‘Great networking opportunities and sharing ideas and CPD activities’ 

e ‘Sessions run by the leaders at Secondary School X were quite useful, | but we learned more the 
collaboration with other schools.’ 

e ‘Good opportunity to share ideas and good practice’ 

e The local workshops allowed us to make connections with other schools outside of our trios, though | 
am not sure they were always a good use of time 


‘A lot of the workshops were more refreshers as we as a school already implement these things. There 
were some useful examples of how they could be used. Though they were interesting and did provide 
some good CPD for the staff, probably the most beneficial thing we got was from doing CtG in the 
school’ 

e ‘The times when we talked with other teachers were the most useful. Some of the presentations were 
not really relevant, but | think everyone appreciated the time to meet’ 

e ‘don’t know why all the people from the school were not at the same workshops, because it is hard to 
find time for meetings in school’ 
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Summary 


The local workshops were generally well received, and clearly attracted more positive responses from 
the respondents than the earlier round of the programme when workshops were located regionally and 
often quite distant from participating schools. All respondents reported that these events were positive 
experiences, and features that attracted particular approval were opportunities to meet with colleagues 
from other schools, and to hear about what challenges they faced and how these were being tackled. 
(Some) inputs were also valued, though the most useful ideas seemed to stem directly from the 
interactions that took place with other participants. 


It seems however that workshops may have been structured differently in different clusters, with some 
respondents commenting on how useful it was to be out of school together as a programme team, while 
others asked why they could not have shared workshops, specifically so this could happen. There were 
other irritations for some- the balance between inputs and activities for example, or the requirement to 
fill in report forms which were collected in and sent to the national CtG team- several respondents felt 
that as this activity produced no feedback it was not a good use of time, though it could serve as a 
stimulus for reflection. 


Q14. In terms of impact, how useful/important do you feel working within a trio 
of schools was? 


Very useful 


They had some 
influence 


Not very useful 
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‘We worked better with one school than the other, but this is definitely a good feature, and we shared a 

lot with one school’ 

‘This was the best feature. We will stay in contact and work together again, I’m sure’ 

‘The best part, really enjoyed it!’ 

‘We have been working as four schools and this model has worked very well for us’ 

‘Sharing ideas was very helpful, and seeing that other schools have the same problems is reassuring. 

Actually we worked separately, but somehow having each other there was important’ 

e =‘ think that working with colleagues from the trio is the best staff development | have had in thirty odd 
years of teaching...’ 

e 6 ‘Great experience. We may not be in the programme next year, but we will definitely continue to work 
with the other two schools...’ 

e ‘For me this was the best thing. We will continue to collaborate even if we are not all in Challenge next 
year’ 

° ‘We worked closely with one of the schools, but the other did not seem so involved’ 

e ‘Just as with the trio work, the impact on trio schools depends on the drive and willingness to accept 
guidance of the school team.’ 

e = ‘The first year in the programme was quite isolated; very little JPD took place. In the 2nd year, there 

were far more opportunities for this. The third year was disappointing to begin with as 2 schools from the 
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four in our group pulled out and therefore the JPD was just a duo. It is the same scenario this year. It is 
useful but the networking in the local workshops is more effective.’ 

e ‘A Secondary school would be useful in every trio- we were all Primary schools and maybe we are too 
similar’ 

e ‘Being the only Primary within a trio with 2 Secondary schools, it was difficult to come up with mutually 
beneficial activities for the age of the children. We became more a 'helper' in helping the other schools 
do what they wanted to do.’ 


Summary 


There was very strong support for working within trios of schools, with several respondents identifying 
this as the most useful dimension of the programme, and citing benefits they had received from this 
close working relationship which brought together those actually involved in the programme delivery in 
the different schools, rather than just the senior managers. Several schools expressed the view that 
relationships forged with other schools during this programme would continue well beyond it. At their 
best, the trios functioned as a brokerage for the exchange of information and approaches, and a space 
where knowledge and skills could be developed within the practical context of programme activities, 
offering what were considered excellent staff development opportunities. 


Of course, things did not always run smoothly- on a few occasions the trios broke down, either because 
a school pulled out or because cross-phase relationships are harder to develop and sustain. However, 
even where this happened, this tends to be attributed to specific relationships problems rather than a 
failure of the ‘trio’ arrangement as such; generally the schools seem to feel that investment in the trio 
brought dividends. 


Q15. In terms of impact, how useful/important do you feel the collaboration with 
colleagues within the school was? 


Very 
useful/impor... 


It had some 
influence 


Not very 
important 
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e ‘You need the senior team to support for the time out of school and the paras support the students all 
the time, not just in the class’ 

e ‘Working between teachers and paras was very helpful. We did not really have a lot of contact with 
school leader once we got going’ 

e ‘For the project we attached TAs to English and maths, instead of students. This was very helpful, 
because it made us plan together’ 

e = ‘It is always highly valuable to work in cross department teams to share experiences, weakness and 
strengths and then share with departments.’ 

e ‘Collaboration between colleagues is key and we have worked strongly together, making collaborative 
decisions and as lead | have myself been involved at ground level and not just as a facilitator. This has 
raised the profile of the group’. 
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e = ‘Really crucial in how the programme progresses as practitioners learn from one another. Carousel 
model really well received as sharing information between schools’ 

‘I think that if you ask the paras they will say they felt much more part of a whole school team..’ 

‘the team in the school worked well together, but we usually do anyway’ 

‘We work together all the time’ 

‘But then we always do collaborate’. 

‘Nothing was followed up outside of the workshops’ 


Summary 


Respondents were mostly positive about the levels of cooperation in school, with a number of comments 
about both the importance of senior leadership involvement with teachers and the integration of 
teachers and teaching assistants within the implementation strategy. Perhaps this underlines that while 
it is often taken for granted, internal communication and cooperation within schools is not always as 
effective as it might be. Certainly it seems that a clearly articulated, common agenda with thought 
through connections between the various leadership, teaching and support roles required to implement 
proved popular with the participating schools. 


Q16. What use did your school/trio make of resources and activities presented 
during workshops? 


These were 
used regularly 


These were 
used... 


We made little 
use of these 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


‘Mostly we did our own thing. But ideas can be useful to start from, even if you don't use them’ 

‘Some of the ideas were good, but mostly you have to alter them to use them in school.’ 

‘Early on these gave us some ideas, but nothing you could really take away and use’ 

A lot of the resources are now in use across my school, question matrix and academic word list. Our 

partner school gave a good choice of ideas at their maths workshop 

e Weare using Talk for Writing resources with our group and have utilized ideas from carousels. We have 
used the Summer Holiday box ideas for the last three years. 

e ‘Many of things we heard about are now implemented, one way or another’ 

e ‘think that time is a problem. We heard about a lot of things that were really interesting and might be 
useful, but you only have so much time to develop new things, so we did not make as much use as we 
could have’ 

e = ‘When | think about it we did not use much apart from the original project goals planning sheet exercise’ 

° ‘It was useful to review things from time to time, but sometimes the tasks we did were a bit abstract, not 
really related to what we were doing in school’ 

e We spent too much time filling in forms about what we had done. | would have liked to hear more about 

what other schools are doing’ 
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Summary 


The responses to this question were more evenly spread; some schools made regular use of ideas and 
resources originating from the workshop sessions, some used them from time-to-time and as they 
seemed appropriate, while some made almost no use of these. It seems that resources were more 
likely to be picked up if they were introduced in presentations from other schools in the cluster, than if 
they were templates from the CtG central team. This is not really surprising; teachers are essentially 
practical people and knowing that something has been used and works is more likely to influence them 
than prettily produced templates. There seems to be a widely held belief that almost nothing you can 
get from a book ‘works’, without the investment of time and energy to ‘customize’ it for use in a school 
context. Consequently, things others have already trialled and refined are likely to be preferred. 


Q17. What use has the school/trio made of resources available on the CtG 
website? 


We have made 
use of a num... 


We have made 
use of one o... 


We have made 
little use o... 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


e ‘Some but not all- growing as website has been updated’ 

e ‘have used a few as leader but | am not sure how regularly the team check website- the emails that get 
sent are generally very useful’ 

‘For example the pupil tracker is now used for all FSM at the start of each autumn term’ 

‘We are using the pupil tracker and one or two other bits...’ 

‘We have adapted materials but not used materials direct.’ 

‘It is about time really. | think there is probably a lot of good stuff there but | have not really looked 
myself. One of the other schools did more, and told us it was useful.’ 

e = ‘Not really looked at this, too busy..’ 

e ‘I did look once or twice, but | don’t think it is updated very often’ 

e ‘None really- we get copies of all the documents we need at the workshops’ 


Summary 


The small proportion of participating schools claiming to make active use of the CtG website was 
underlined by the smaller number of comments this question attracted. While it seems that one or two 
schools have used the website as a potential source of ideas, there are no clear examples of the website 
contents finding their way into the schools’ strategies and practices. It would appear that schools visit 
log onto the site only relatively infrequently, and indeed on checking the site one can see that the CtG 
team may be well aware of this, as the site itself does not seem to be updated very frequently either. 
However, the fact that this was not mentioned by any of the respondents reinforces the notion that it is 
not seen as a major source of information or resources for the programme. 
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Q18. What problems, if any did you encounter when implementing the 
programme in school? 


e ‘None really, once we knew what we were trying to do’ 

e ‘Finding time, and being out of school for visits- it is great to be able to take the kids out or see what 

other schools are doing, but at the same time you feel guilty about the classes you are not teaching’ 

‘Getting cover for meetings. Finding time to meet with colleagues in school’ 

‘Remembering and having time and space to implement it myself rather than just leading it’ 

‘Rooms to deliver training in, as most schools do not have lots of room available!’ 

‘Just time and resources’ 

‘Some problems arose in the second year because myself and my teacher were not based with the 
year group we targeted, it was more difficult to build relationships with these children’ 

e ‘Some staff members have been unhappy that pupils are removed from lessons for group meetings and 
activities’. 

e ‘Time was an issue. As a Primary school, being in a trio with two Secondary schools was a bit difficult as 
well’ 

e ‘Just cover, there is never enough cover!’ 

e ‘Setting meant the students were not together in class as often as we would have liked, and chasing 
them as individuals took a lot of time. Not enough time to visit other schools. Pressure to get results 
quickly’ 

e = ‘Additional paperwork, and intervention base in school drew children away from class teachers’ 

e = ‘Resistance at first to the ‘special treatment’ of one group of PP students, but acceptance over time, 
once understanding was built.’ 

e ‘Hard to prioritize just this group of children when the same strategies are required throughout the 
school.’ 


Summary 


The problems reported were mainly to do with resources, organisation, and teacher beliefs. Of the 
resources, it was time, both directly and indirectly that was causing the most difficulty. There was not 
sufficient time to get together with colleagues in the school, to plan activities. There was not enough 
cover available (bought-in time) to allow all the visits to other schools that would have been useful. 
Attendance at workshops created further time pressures. 


Organisational difficulties were mainly related to getting the target cohort pupils together in school. As 
previously mentioned, setting and timetabling meant that often these children were distributed across 
several classes. It was not easy to extricate them for joint activities, and when it was possible, there 
was the problem of what they were missing to make it possible. 


In a minority of cases, there were also some doubts about the strategy underpinning the CtG approach. 
While it could be understood that targeting small groups of pupils makes sense when there are only a 
small number of PP pupils in the school, some teachers clearly had more difficulty accepting that the 
same approach was sensible when a majority of pupils qualified for PP support, and those who did not 
were so close to the line that it made little difference. 


Q19. What would you say are the most important things that you/your school 
have learned from its participation in the CtG Programme? 


e =I think we always knew that we could make a difference to some children, but this gave us a particular 
focus on those who really need help’ 

e = ‘Paras can make a big difference if they are used properly. Cooperation with other schools helps all of 
us to develop’ 

e ‘Some of our most vulnerable learners could be helped more if we target resources and interventions 
better’ 

e = ‘Individually, how to lead and project manage. And for the school, working together with colleagues to 
share and embed good practice’ 

e ‘Crucial need to base interventions on research and IEP, especially meta-cognitive strategies Making 
the model suit the school and needs of children Promoting an ethos of attainment. Understanding the 
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barriers to learning that confront disadvantaged children. Need to better understand data. Clear 
responsibilities in the programme delivery’ 

e ‘Teachers can make a difference, so can TAs when they work closely with teachers’ 

e ‘JPD with other schools works. There are barriers limiting the learning of PP students, but these can be 
overcome. New ideas for teaching and learning strategies.’ 

e ‘Most importantly, the PP pupils are capable of achieving in line with their peers if intervention is put in 
place early enough and consistently followed. Parental engagement is key to the progress of these 
pupils.’ 

e ‘think we have a much better understanding about real CPD. You can do it all the time and you can do 
it for yourself.’ 

e ‘The importance of collaboration with colleagues, all colleagues, not just other teachers.’ 

e ‘Developing literacy needs a strategy that goes beyond teaching English. Many of our students have 
very limited life experiences. TAs are very important to some students.’ 

e ‘What we are implementing for our PP children is along the ‘right lines’. 


Summary 


A range of learning outcomes for participating schools and teachers were mentioned. First, these 
related to pupil potential and teacher expectations- several respondents referred to increased 
expectations of what disadvantaged pupils are able to achieve given appropriate support, and to 
increased awareness of the powers teachers and teaching assistants have to bring this about. Linked 
to this was a feeling that the programme had prompted staff to look more closely at the particular needs 
and the barriers to learning that often reduce PP pupil attainment. A clearer understanding of these 
factors enables staff to make better decisions about and offer better provision to these pupils. 


What can be learned through collaboration with colleagues, both within and across schools was also 
commented on; it is clear that collaboration with peers is an important professional development 
opportunity, and one that has been highly valued by participants in these schools. Improved 
relationships, more effective teamwork and better utilisation of teaching assistants were also 
mentioned, suggesting that during the programme schools learned quite a lot about fairly simple things 
that they could do better, including clarifying goals and roles. 


Q20. Can you suggest any changes to the CtG programme that you feel would 
make it more effective in future? 


e ‘Less workshops, more work with partner schools. Groups of schools that are closer together’ 

e ‘More meetings with trio, less workshops. Less paperwork to fill in at the workshops’ 

e ‘Fewer workshops. Workshops where we are all together from the one school .Lower fee for Primary 
schools!’ 

e ‘Considering different models for collaboration, away from trios’ 

e | don't think we need all the meetings, better to meet in school more often?’ 

e ‘Maybe to consider how trio members back in school can be contacted more easily- it is quite hard to 
stay in touch between meetings.’ 

e ‘More opportunities to talk as trios, as this is so valuable as sometimes there's such a packed 
programme that little opportunity for discussion is offered.’ 

e ‘Make sure that schools in the trios are from the same phase rather than a mix.’ 

e ‘Leads, teachers and TAs attending sessions together’. 

e ‘Alot of admin was required and no feedback was provided on all the forms we filled in. It felt like doing 
paperwork for the sake of it.’ 

e ‘think that we could meet more often as trios, maybe in each other's schools, and not as a local group 
so often. | think that practical examples are more useful to us than lectures about what the problems 
are.’ 


Summary 


For the most part, the suggestions put forward in answer to this question echo comments made in 
answer to earlier questions. The preference for in-school meetings and trio meetings over workshops 
is repeated, as are the activities presented at workshops, with less ‘lectures’ and ‘form-filling’ and more 
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interaction between participants urged. The proposal that whole school groups attend workshop 
sessions together is also raised again. 


More opportunities to fully exploit the potential of the trios are also recommended- this was seen as the 
most important element of the programme, and it is clear that many feel that their schools could have 
learned more from each other if there had been more time for cross-school visits and discussions. 
Inevitably, liaising between schools, and between different levels at schools, is also an issue for some. 
There is also a feeling that more thought needs to be given to how teachers can keep in touch with 
each other, and how teaching assistants can keep in touch with each other, rather than simply 
maintaining contacts between school leaders. 


Q21. Would you recommend the CtG Programme to other schools? 


Definitely 


Possibly 


Unlikely 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


e ‘I would tell them it is hard work but we hope it will be worth it’ 

e = ‘If you get a chance, go for it!’ 

e ‘Overall it was a good experience. Yes, | think most schools would gain something from taking part’ 

e = ‘It is a great way to share and gain practice from other schools’ 

e ‘Avery informative experience. | think that the pupils got quite a lot from it and so did the staff, so you 
have to say it is a worthwhile thing to do.’ 

e =‘ have learned a lot from being involved in this programme’ 

e = ‘Certainly | would recommend it, but you have to decide what your own priorities are’ 

e ‘Depends on the number of Paraprofessionals within a school. If a large number, no.’ 

e = ‘It takes a lot of time, and you need good TAs to get the best out of it’ 

e ‘It costs quite a lot. For a small school like mine that is a big issue. If you have the money it is fine, but it 


depends on what else you could do with the money. 
e =‘ think we all feel it has been worthwhile, but he head says that it is too expensive to continue so we 
wont be doing it again next year’ 


Summary 


Almost three-quarters of the respondents said that they would definitely recommend the programme to 
other schools, with a further 20% saying they would ‘probably’ recommend it. There is no doubt that, 
overall, the great majority of schools within the programme feel that it has been worthwhile, that they 
have learned from it, that their targeted pupils have gained from it, and that in due course the benefits 
will spread out to reach greater numbers of pupils, both PP and non-PP. Of course, they do not present 
overwhelming evidence showing that this recommendation is based on solid, verifiable pupil outcomes. 
Rather, they point to modest improvements, or to what they consider to be variables leading improved 
performance, such as attitudes to school, and behaviours such as improved attendance. 


Education Endowment Foundation 57 


Challenge the Gap 


Additionally, they have reported many other benefits- enriched pupil experiences, enhanced pupil self 
esteem, high quality professional development opportunities, increased awareness of possible 
strategies and approaches, raised expectations of staff and pupils alike, better motivated and more 
purposefully used teaching assistants, improved leadership and teamwork within the school- which they 
feel have derived from participation in the programme. These so-called soft measures may not be 
immediately reflected in any narrowing of the gap, but common sense argues that they do build healthier 
schools, and that healthier schools are better placed to serve the needs of all of their pupils. Inevitably, 
the costs of such a programme need to be considered, both the financial costs of participation and the 
opportunity costs of programmes and activities foregone in order to do so, but on the basis of the 
responses to this survey, it would appear that the majority of the schools responding believe it to be a 
wise investment. 
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Appendix 2: Case study consent and invitation letters 


Evaluation of the Challenge the Gap project 


Protocol for initial visits to trios of schools 


Contact : Professor Mel West 
School of Education 
University of Manchester 
Manchester M13 9PL 

0161 306 1302 


Email: mel.west@manchester.ac.uk 
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Introduction 


Our aim is to carry out the evaluation in a way that will contribute to the development of the project 
whilst, at the same time, collecting and analysing evidence that will enable us to provide a credible, 
independent judgment as to its effectiveness. With this purpose in mind, we are using a two-strand 
approach, focusing on both impact and process. The impact strand is analysing outcomes in 
performance associated with being a member of a school-to-school collaboration, whereas the 
process strand seeks to investigate the nature of changes in school/classroom practices. 


This protocol outlines the proposed approach to the initial round of interviews with colleagues from 
participating schools. In this first tranche, we will visit three trios of schools to gather information and 
perspectives on the early processes and activities within the project. This material, together with a 
further series of interviews to be conducted towards the end of the school year, will contribute to 
accounts of how the project has developed in these schools during the school year 2012-13. 


Collecting information 


Broadly stated, our engagement with the schools will be guided by the following questions: 


1) How/why did the school become involved in this project? 


2) What arrangements have been in school in order to implement the CtG approach? 


3) Are there factors that may mean these arrangements difficult to sustain? If so, how are these 
being tackled? 


4) What aspects of collaboration are proving to be most powerful in mobilising efforts within the 
school? 


During this initial phase of our involvement we will use as our starting point the summaries provided 
by each school in relation to the three sections of the planning tool provided by the project team. This 
has required schools to focus on three elements, as follows: 
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ee 


What do we intend to do What changes do we What progress in 

to produce an impact on expect to see that will attainment do we expect 
pupil progress and eventually produce an to see at the end of a 
attainment? impact on attainment? term? 


What progress in 


What evidence can we attainment do we expect 
gather for this? to see at the end of the 
year? 


Sample 


In discussion with the project team, we have chosen three trios of schools, in different parts of the 
country, where we will develop accounts of what happens in each of the partner schools. During the 
year we will visit all these schools, at least twice, to collect information on activities arising from the 
project and to interview a range of staff, including some who have been less directly involved in the 
project strategies. 


Towards the end of the year we will prepare an account of practice for each trio, and these accounts 
will be shared with the particular schools. This will allow us to test the credibility of our analysis with 
stakeholders, whilst at the same time offering feedback that should promote reflection and 
professional development within the schools. 


The initial visit 


Our first visit to the schools will take place in January or February 2013. During this visit we would 
like to hold discussions with the project leader, plus at least one teacher and one member of the 
support staff who are closely involved in the project. In addition, we will be happy to meet other 
colleagues that schools wish to nominate, individually or in groups. 


The conversations will be guided by the questions set out above. The approach will be informal. 
Colleagues will be encouraged to focus on issues that seem to be of most relevance within the school 
and across their trio of schools. 


We would like to tape record the discussions. This is so that we can replay them later as we attempt 
to develop an account of what has happened so far. The names of individuals will not be included in 
these accounts, although inevitably it will be relatively easy to guess who said what at this stage. 
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The evaluation team 


The team brings together a range of experience and expertise in relation to the themes of this 
particular project, as well as varied methodological strengths. The team members are as follows: 


Professor Mel West. He has carried out extensive research in relation to networking and 
collaboration, within and between schools, both for NCSL and for the DfE during the Leadership 
Incentive Grant initiative. He has a particular interest in aspects of management and leadership. 


Professor Mel Ainscow. He is internationally recognized for his research around the themes of 
inclusion and equity. He was also involved in the LIG implementation and subsequently as the 
Government’s Chief Advisor for the Greater Manchester Challenge, where extensive use was made 
of school-to-school collaboration. 


Dr Michael Wigelsworth. He has been involved in evaluations measuring the impact of a series of 
national initiatives, including Primary SEAL, Secondary SEAL and Achievement for All, all which have 
involved the analysis and interpretation of impact from specific interventions on national data sets. 


Dr Trudi Martin. She has extensive experience in the implementation and evaluation of learning and 
development programmes, and is also a development coach. Her particular interests are the provision 
of education for children who have complex learning difficulties and knowledge sharing within and 
between schools. 
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Skeleton Invitation to Participate (Intitially sent to CtG School Heads and Project Leads) 


Dear xxx 


| am working with the team from the University of Manchester on the national evaluation of the 
‘Challenge the Gap' project, commissioned by the Education Endowment Foundation. Under the 
evaluation contract, were invited both to monitor the relative progress of 'Challenge' cohort pupils, and 
to produce a number of cases studies of 'Challenge’ schools that illuminate the implementation and 
impact of the programme at school level. For your information, | attach the protocol for school case 
studies agreed with EEF and the Challenge the Gap team when the first round of case studies took 
place. 


We are now seeking to visit a sample of schools that have been put forward to us by Challenge 
Partners in response to our request for a list of schools where the programme is being implemented, 
to gather your reflections on the school’s/your own involvement over the past year, and to discover 
what factors you feel have enabled the school to successfully implement the programme. Of course, 
we would also be interested to hear of any actions that you feel might be taken to improve the 
programme as well. We are looking to have a series of conversations with school leaders and others 
who have been closely involved in the implementation, which will be used both as evidence of the 
impact of the programme on the school and the progress of target pupils, and to help us develop a 
questionnaire about implementation strategies and issues that all participating schools will be invited 
to complete in due course. 


| am hoping therefore that it would be possible to arrange a visit to your school to carry out interviews 
with you and perhaps a few of your colleagues who have been directly involved in the programme. the 
interviews will not be over-long- somewhere in the region of 45 minutes in each case, and as they are 
solely intended to inform the evaluation, data collected in this exercise not be published in any way 
that identifies the school or individuals without prior consent. 


| hope that you will feel able to take part in this activity, which a major component of the overall 
evaluation of the programme that was agreed between the Education Endowment Fund and 
Challenge Partners. 


| have some flexibility regarding dates and hope to be able to fit in with what suits best though | am 
hoping to carry out the bulk of these interviews during (INSERT DATES). Please feel free to suggest 
possible alternative dates if the times | have suggested are not convenient. Ideally, | would like to 
spend a day in your school, talking to the project lead, and to some teachers who have teachers and 
to one or two members of the support staff who have been actively involved in the 

project. Participation is voluntary, but | hope that you and your colleagues will take part. 


If you are happy to be involved in the evaluation of 'Challenge the Gap’, please let me know and | will 
be in touch nearer the time to arrange a convenient time to visit the school. In the meantime | hope 
that you have a pleasant summer break. 


With thanks and kind regards, etc 
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Appendix 3: Extracts from school case studies 


As noted in the report, some 17 case studies detailing the experiences of schools that participated in 
the Challenge the Gap Programme were written up. The sample included schools drawn from across 
the first three years of the Programme, and so they straddled the transition from a subsidised pilot into 
a participant funded school improvement programme. The original intention was to publish these case- 
studies alongside the report. In the end, however, sheer length makes this impractical. Consequently 
we have attempted to give a flavour of the content below. 


In selecting text that might best convey the experiences of those most closely involved in the 
implementation of the programme within their own schools and trios, we have been guided by two main 
considerations. The first of these comes directly from the format used to write up the cases. When we 
had gathered data from the first batch of schools, these were scrutinised to see what themes or 
elements seemed to recur in the accounts we had, and these became the main organising categories 
for the case-studies. This resulted in fairly broad categories which were then used to structure the 
accounts; motives for joining the programme, the selection of the student cohort, views on the 
programme of those staff members most closely involved in implementation (especially with respect to 
perceived impact) and any particular problems or limitations they had encountered. These categories 
are used here too. The second consideration was to include, as far as possible, the actual words of 
those who were interviewed while the case-studies were being compiled. Again, we have sought to do 
this below 


Why join the programme? 


The proportion of FSM students in the school is currently over 60% of the total. The students targeted 
for the Challenge programme are a mixed group of Year 8 and Year 9 students, who were not 
characterized as a single homogenous group, but rather represented a ‘wide variety of situations and 
problems’ The project‘ ..came at a good time for us, we have done a lot of interventions and a lot of 
things around narrow the gap, but maybe because they are a majority here we have not previously 
given focus to FSM students. (School B) 


Ten years ago, the school successfully reduced a gap in attainment between boys and girls of 20% 
‘simply by focusing on them; making sure all the staff were aware of who the boys were, what their 
target grades were and kind of having a "no excuse" policy, both with staff and students’. The school 
believed it could achieve the same with FSM pupils. (School C) 


‘....the school’s intake continues to come predominantly from disadvantaged sections of the community, 
and, as OFSTED also noted, students joining the school in Year 7 are, on average, two to three terms 
behind their counterparts nationally. The school prioritizes the development their skills in reading, writing 
and mathematics to enable them to learn in other subjects. (School D) 


The particular students targeted have been identified through scrutiny of this cohort (and the elimination 
of children who also have special educational needs) to pick those who seem to be under- performing, 
and can potentially benefit most from the intervention. The school has described the key priorities 
associated with this group as: the need to ensure these pupils are not ‘missed out’ within the range of 
intervention strategies adopted; the need to ensure that these children are set targets that reflect 4/5 
levels of progress during their time in the school ;the need to ensure that the relatively low levels of 
cultural capital found among parents does not hold this group back; the need to ensure that these 
children are not ‘overlooked’, even if they are making ‘expected’ progress. (School E) 


At that time the school was receiving pupil premium funding for 59% of the students, and so had 
focused very hard on raising attainment and achievement within that cohort’. Despite these 
interventions, the school still had a sizeable attainment gap; ‘it was 20% last year — it actually grew 
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larger than the year before — our pupil premium kids did better, but our non-pupil premiums did a lot 
better’ (School J) 


The pupils selected ‘ranged from Reception up to Year Six’. In order to identify this group, senior leaders 
involved ‘spoke to teachers and looked at where the gaps were in our learning, and chose the target 
pupils based on that and on what we thought we could do for them. We also looked at the bigger picture; 
we didn't kind of isolate it — we picked those children on whom we felt we could have a bigger impact 
straight away’ (School kK) 


The school was interested in participating because: ‘We are always on the look-out for ways to 
support the children, firstly cause of our intake; we have an intake where half of our children are on free 
school meals. Lots of very needy children, and we are looking really closely at closing the gap, and 
looking at raising the levels of progress, so if we're looking at that anyway and somebody comes to see 
us saying there's this project that might help you do that, inevitably we wanted to be involved. Because 
we are constantly looking for new strategies and new ways of doing it ... you know, our gap is narrow, 
but we're still trying to focus on those particular children who we think could, we could crack it with.’ 
(School! L) 


This is a large school with a three-form entry and almost 40% of their pupil intake is Free School Meals 
(FSM) children (though the school feels that many more are eligible, but for a variety of reasons do not 
claim for this provision). In addition, around 95% of these pupils have English as an Additional Language 
(EAL). As such, they wanted to ensure that the significant funding they were receiving from Pupil 
Premium was being spent effectively and in a way that could promote tangible outcomes for pupils. 
They also had concerns as a school that too many of their pupils were not developing the capacity to 
learn independently, and were keen to address this by supporting their pupils to be become more self- 
sufficient and confident learners, instilling within them the attitudes and skills that are needed for lifelong 
learning. (School S) 


At the time of joining the programme (The School) felt that there was probably not enough awareness 
among staff about the FSM students, who they were, what were their learning needs; so these were 
not always being addressed and it was easy for FSM students to ‘slip under the radar’. Further, it 
seemed likely that there were other children in the school eligible for FSM, whose parents were -for 
whatever reason- not claiming them. The school was therefore very keen to develop a way of tackling 
this problem without stigmatizing FSM children. (School T) 


‘The decision to join the Challenge the Gap programme was taken quite late in the day, and without a 
great deal of discussion in the school about the purposes and structure of the programme or the 
practical arrangements that would be necessary in order to participate fully. The headteacher then 
attended an initial planning meeting with other interested headteachers, and was perhaps surprised to 
hear what exactly was involved’ (School V) 


Selection of Student Cohort 


Each cross-school trio has developed its own point of focus. This followed an initial period of ‘feeling 
around in the dark regarding what the programme entailed, though ‘it really started taking off once they 
had decided it would be possible to pursue different projects in each school, depending on local 
priorities. (School A) 


In maths, links are being made using the school’s specialist sports’ college status, to establish contacts 
with clubs and develop activities as a vehicle for maths education. This is supplemented by a number 
of specific skills-teaching interventions, progress coaching and some additional maths lessons. Literacy 
skills are being targeted through additional teaching sessions and progress coaching too, and through 
a programme of practical activities and trips that promote reading and writing skills. (School B) 
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Each trio had its own focus, most of which were inspired by ideas from the Challenge workshops. For 
instance, the leader's was 'raising awareness’ and ‘attainment’. When she met her two trio members, 
they would bring the data tracking the FSM students in the subjects they line manage. Two trios focused 
on feedback in their curriculum areas. Many of the paraprofessionals focused on increasing the 
students' levels of confidence. (School C) 


The specific cohort targeted in the project were selected from Year 9, and includes 18 students who 
present a range of difficulties, including challenging backgrounds, poor behaviour and low self-esteem. 
This group is the focus for the in-school monitoring and cross-school activities, but the general principles 
underpinning the Challenge match the school’s philosophy, and the ways of working with this cohort 
have been adopted across the year group. (School D) 


The pupils were selected ‘With the team leader for Year Eight at the time, based on a combination of 
the data, paying particular attention to those not making expected progress in Maths and English. We 
had originally thought that would mean working with those with learning difficulties or challenges. Now, 
we find that it is also our able pupil premium students who are also vulnerable. Some of them have 
attendance issues, though nothing too chronic. Some of them have issues around parental 
engagement, or they have a lack of experience of Higher Education. They’re often the ones just not 
quite knowing what to do with their lives’ (School J) 


In identifying the target cohort, the school placed special emphasis on literacy, as ‘it definitely has a 
bigger impact down through the year, especially where we concentrated on speaking and listening skills, 
because without that we weren't going to be able to move anything else on’. (School K) 


What distinguished the phonics work done as part of the programme from that done as part of routine 
classroom practice was that it was now taking place in specific, supported intervention groups, ‘so that 
they were now working with similar children. We trained our staff for this, we looked at the teachers 
who were working with those children and ensured there was sufficient training for those teachers and 
teaching assistants, so they knew what they were trying to do and were confident’. (School K) 


We sat with the data for every single year 8, and we looked at effort grades ... we looked at their levels, 
we looked at how they've come into us; what they got for KS2, what they're predicted, where they were 
on that prediction, the amount of behaviour problems there've been - or not - and from that we pulled 
out our fifteen. So there were, in a way, different decisions for different children, almost, so obviously 
they were all on free school meals. But in one or two cases they were children who were presenting, 
who were, who had potential, but who were presenting behaviour difficulties who we wanted to nip in 
the bud before it became a full-blown problem. One or two of the other children, it was, there was no 
real behaviour problems, they were all right in terms of effort, but really, they really weren't achieving. 
So, we arrived at our fifteen from all of those directions’ (School L) 


‘The school has identified a target group of 12 under-achieving students in each year group, with whom 

they work to improve attitudes and educational outcomes through enriching these children’s 
experiences in a number of ways. All CtG students have specially designated ‘achievement 
coordinators’ in the school, with whom they spend three 40 minute sessions per week discussing their 
progress and any problems, where they have the opportunity to catch up with work in the core subjects 
and also, on a one-to-one basis, discuss their learning and any difficulties and challenges they may 
encountered have with a trusted teacher. During this time, the students also occasionally complete 
surveys designed to capture their feelings and perceptions, and have meetings with their pastoral 
mentor, to discuss how they feel about their learning, but with a focus on the social and emotional 
elements of their educational experience.’ (School T) 


‘...finding a focus for the school’s own activities was not difficult. Despite its small size, the school has 
two classes in year three for whom writing is an established area requiring improvement. It did not take 
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long therefore to agree that literacy, and particularly a cohort of year three students, would be the 
appropriate target group’. (School W) 


Staff Comments 


‘It’s been a real journey for a lot of the students. | was impressed... [by] how the students were so aware 
of their progress; not just on a grade level, it’s not just about saying “right, we’re going to get you a 
grade”, but actually about life experience and life chances ... in life, they feel they definitely have things 
open to them that weren't open before’. (School A) 


‘We are emphasizing the building of cultural capital, this is not just about accelerating progress, it is 
about raising aspirations and expectations, increasing self-awareness and life skills. We need to extend 
the horizons for these children.’ (School B) 


‘ _..the nature of the cohort has made us realise that it is not just about attainment- it is about access 
and aspiration too. We need to build up social capital for these children, academic focus needs to be 
supported by the development of social capital that enables them to make use of additional learning.’ 
(School D) 


Teachers described the goals of the Challenge variously as ‘maximising progress’, ‘raising hopes and 
expectations’ and ‘broadening experience and developing social skills, the thing these children are 
lacking’. (School E) Initially, different schools were using different year groups so we did an overall plan 
of what we wanted to see throughout the three schools and how we were going to work together to do 
something similar which was good and has meant the data from all three schools is more meaningful. 
We've also created Planning Progress Outcomes to identify what the leaders, teachers and 
paraprofessionals aiming to achieve and we’ve all worked with this document together.(School E) 


‘(Non-teaching staff)...had especially enjoyed meeting colleagues with different roles during the training 
workshops. They had noticed that many of their colleagues were, for example Learning mentors, who 
seemed to have roles which offered more opportunity to influence student attitudes and behaviours. 
They welcomed the opportunity to extend their own roles through the programme, and to develop both 
closer and wider involvement with the FSM target cohort. (School E) 


‘One of our students, she got 5c at Key stage three. So she is one of the students that from the Raise- 
on-line data we are concerned about the amount of progress they make. So that would be part of the 
reason why she was selected, was because she was a 5c in both English and Maths. And, as a student 
she is quite nervous, and she kind of goes under the radar a certain amount. And the more you talk to 
her one-on-one, the more you realise how nervous she is; she's quite shaky. And I think having 
individual attention and a bit more visits and one-to-one and parents coming in; all those little things ... 
has been really, really good for her. For our Rudolph visit, she wrote an article for the school newspaper 
and | worked with her to kind of get that to where she wanted it to be. | got her to talk to the member of 
staff who publishes the newspaper and to put herself out there. And | think her confidence is just 
increasing by increments. And the fact that we know about her, and we know, for instance that she 
wants to be a journalist means that we're putting opportunities her way, and she's raising her hand and 
getting involved more. And she's had, of course, academic mentoring as well; she happens to be one 
of the ones who's worked with me, so she's got targets around that, kind of, participating more, asking 
more questions, being more visible, to kind of aim at developing her confidence a bit more, her 
participation in her education. | think I'm seeing very real changes in her. Becoming more and more 
confident and less and less shaky. So that's one we're really proud of' (School J) 


‘I think we had too many individual things going on. So if you look at the TAs, well two of them had a 
phonics focus and then the other one had Project X, so instead of everyone having the same kind of 
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focus, they had different concerns. Although, | don’t know how differently we could have done it, 
because being in the situation where half the school is one form of entry and the other half of it is two 
form entry, that’s something we found very difficult to plan for- it is extremely hard when you have 
different size groups to deal with in the different years, so you end up doing different things....but still 
we were probably trying to do too many things at once’. (School kK) 


‘mainly they have had mentors who have met with them individually, talking about issues to do with 
their learning’. \nitially, ‘the teachers were meant to be mentoring the children, but that didn’t work at all 
because they didn’t have the time, so we’ve actually got the TAs mentoring the children now which is 
actually really useful cause they havent got tutor groups, they can be free sometimes when the teacher 
can’t’ (School L) 


It’s also good to work cross-phase, Primary and Secondary, because it’s always good to see where the 
children are going ... and for teachers you can get very insular within your phase and so anything that 
encourages Primary schools and Secondary schools to work together can only benefit the children 
because it gives teachers a greater understanding of where the children are coming from or where they 
are heading to, and hopefully it helps Primary prepare better for Secondary and in the same way 
Secondary [teachers] can understand where children get to in Primary and how things work’ (School 
M) 


‘a lot of teachers together, which was fantastic, particularly across authorities, because different 
authorities have different strengths. That’s what really struck me, the collaboration aspect of the 
programme and the opportunities for working together across boundaries — that’s what really, really 
made it’ (School M) 


“It was mainly the TA, without the TA in our school it wouldn't have been as successful, he really 
personalised the pupils’ learning.It was really his baby and he spent a lot of time preparing and planning 
it all... we talked him through the data as he really struggled understanding that — where we needed 
the children to get to, how we would measure it- but he’d do the planning and talk to the class teacher 
about the children.” (School N) 


Challenge the Gap provided them with the opportunity to be involved in something they had not done 
before, working closely with colleagues- both teachers and other non-teaching staff- was probably the 
best professional development they had ever experienced :“There are not a lot of those opportunities 
going round, and this was something that they could be at the core of, a whole school project here then 
involvement with a similar project in another school which was excellent for them to be part of; and it 
will also help their future career....’ (School O) 


‘we've tried to give them experiences they might not have outside of school..... some are quite simple 
things, like going to the shop with money, because many of these children never go to a shop or handle 
money. So it is learning in the classroom that they can then draw on outside of school, and this links 
with a wider school initiative to try to link out of school experiences with the development of vocabulary 
and literacy’ (School S) 


‘For example, in one of our CtG workshops, we did sessions where teachers had to bring in examples 
of how students respond to feedback. This has been a real success in our school and we’ve really 
driven it forward with staff, we’ve delivered a range of interactive sessions on marking and feedback 
during INSET days, regular senior and middle leader monitoring of marking and feedback, and we have 
a whole school strategy whereby the teachers regularly give students time at the start of their lesson to 
read their comments and reflect on them’. (School T) 


‘Normally, when you plan a trip, it is so the pupils can see something, something that will perhaps be 
outside their day-to-day experience. So there is always content, because we want the trips to expand 
their experience, maybe expose them to something for the first time. It might be something many of us 
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take for granted, but still it is missing from the lives these pupils lead. And when you have made the 
trip, the writing and projects you develop from it are a sort of bonus, you try to make use of this 
experience and build on it. | have been doing this for years, planning for the content, but not really 
thinking about the writing. We decided to approach it differently, to plan for the writing. | can’t believe 
that | never thought of this before.... it seems so simple, but it leads to a totally different way of thinking’ 
(School V) 


‘Teaching assistants played a central role in the programme, not just supporting the trips and the writing 
lessons, but often working with children themselves to deliver the intervention. The main TA involved 
felt that she had gained an enormous amount from her participation. ‘/ gained so much knowleage, even 
doing some co-teaching...’. She felt much more involved with the children’s progress, and became 
increasingly aware that she had an important part to play in moving them on academically, rather than 
just supporting them and helping to deal with behavioural issues. ‘/t really gave me confidence, | had 
never been involved in planning..... doing book scrutiny... it really gave me new insights. And the links 
with TAs from the other schools were so positive, we compared approaches, | heard about different 
experiences and training’. (School W) 


Perceived Impact 


‘You do get stuck in your ways, and it’s given me a sort of kick to, you know, develop and think about 
feedback, and it’s not just ticking and saying “well done” but looking for their response, it’s giving them 
time to actually read what you’ve written, making them physically write a response. It’s certainly made 
me think a lot more about what I’m saying and how I’m saying it’ (School A) 


‘...much of what had been done in the school seemed to have had a very positive impact on attitudes 
and behaviour, and not just of the students- the photographs identifying the students involved in the 
Challenge that had been put on display in the staffroom to encourage positive comment had been a 
great success, and had generated lots of positive thoughts that could be shared with parents. Similarly, 
the summer Challenge had been a great success; students clearly enjoyed the activities they were 
asked to carry out, and frequently still talked about these’. (School B) 


'A lot of our teachers feel more empowered. | think they certainly feel that they're being treated more 
like professionals. They quite liked being given the readings to do and | think that it's made them ... 
realise that we can't accept second best from the kids’ (School C) 


The main impact the project has had so far is probably outside its initial remit. ‘it has highlighted to all 
staff the very substantial numbers of FSM children we have throughout the school. Previously, many 
seemed not to notice this, all are aware now’. (School D) 


‘_..the programme was ‘working better than | expected, it has been easier to get off the ground. The 
fact that it can be tailored to you own school, that is the strength of it, we can customize the tools we 
are given, deliver them in our own way..... (School E) 


‘the majority of them have made good progress over the year’. In fact, most progress has been seen in 
reading, both in the target cohort and across the academy more widely, ‘the students were so 
enthusiastic about what they were doing that the other students took on that challenge; they’ve started 
doing things in that way ... So it has spread, it is there, but it’s not solely the free school meal children 
that have done it’. (School F) 


The fact is that we’ve now got girls who feel much more positive about school than they did previously. 
The opportunities we ve provided to visit universities, for many of them for the first time. Also, the training 
has been really good’. It has been ‘really useful to get together with other schools and see what they're 
doing’ (School J) 
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The impact of our work has been astounding. In the first year of activity, most children made two sub- 
levels of progress, and their well-being has improved beyond recognition. Some even visit our learning 
wall independently to get the resources they need to move on. Our pupil surveys show that confidence 
has significantly increased and that reading progress is much higher this year. (School! J) 


‘Because we had looked at our pupil premium children in a different light, we then managed to close 
the gap for some of those learners, children who had been falling behind. We still have a long way to 
go, but what it did was it made us look at things in a different way, and we have put different ideas into 
practice based around what we found helped the pupils last year’. (School K) 


‘....teachers who are leading the trios........ ‘they've been finding some really interesting responses and 
reactions. We're just doing some analysis of assessments, and they're ongoing, being recorded at the 
moment, and | have had people approaching me and talking about children making an interesting 
amount of progress compared with what we would have expected’. (School L) 


‘....encouraged teachers to think of different ways to engage their pupil premium children, for example, 
taking them to a safari park when they studied the topic of animals “so it enhanced their lives and 
increased their learning experiences, which they brought back into school...... and they had real 
experiences to write about, so that engages them more and | think it has improved their attainment” 
(School M) 


‘ 


athe (the coordinator) felt a highlight from the project was that it made them really focus on the 
participating Pupil Premium group as individuals ‘What are we actually doing for these children? What’s 
the impact of it?’ and there were significant benefits for the TA, who really developed leadership skills 
during the year he worked with the pupils ‘he really blossomed, because the ‘owned’ those children’. 
(School N) 


(While the school) is efficient at tracking quantitative data, this project made them look more closely at 
and reflect on soft data: ‘/t was how can we get them to be really switched on to learning? And it is that 
tracking of the soft data that we don‘ really do as a school. Obviously we have conversations as to how 
they are getting on and any intervention that’s needed, any nurture that’s needed to help them out, but 
it was that [soft data tracking] that gave us the clue, so that was important’ (School N) 


“..a mixed bag”. Some students made “ absolutely amazing progress, some made what we thought 
was decent progress, but their attendance declined so that progress was a little bit less obvious. There 
is another case where poor attendance made progress stall. But some made exceptional progress in 
maths — jumping two levels, nearly three levels! A 3C to a 6C, which is not bad going in the space of 18 
months”. (School O) 


‘the impact of our work has been astounding. In the first year of [CtG] activity, many children progressed 
two sub-levels ...... and their wellbeing has also improved beyond recognition ... Additionally the FSM 
pupils originally targeted in the work are now seen as leaders — they have a strong profile within the 
school and are actively engaged in helping others....Some even visit our learning wall independently to 
get the resources they need to move on. Our pupil surveys show that confidence has significantly 
increased and that reading progress is much higher this year’. (School S) 


‘There is a lot more knowledge in the system about what works for disadvantaged students and that is 
being shared between schools and in the press and therefore people fee! more confident that they can 
tackle these issues. For example, in our case we have addressed these issues through marking and 
feedback or small group tuition’. (School T) 


‘...almost all pupils have exceeded their ‘expected’ progress level following the sustained and 
concentrated efforts to improve writing’. Almost without exception, the writing outcomes produced by 


Education Endowment Foundation 70 


Challenge the Gap 


pupils were more substantial, showed more imagination, used a wider vocabulary and demonstrated 
better retention. It seemed likely that this general acceleration in progress would mean that the ‘gap 
between pupils here and national average attainment would close, though there might not be a 
narrowing of the in-school gap at this stage’. Equally importantly, there had been parallel improvements 
in the pupils verbal skills, and some evidence of better listening skills. It seemed likely that the full impact 
of the programme on the children’s writing had not yet been realised, and this would continue to develop. 
Already there were signs that, ‘for some, self-knowledge had increased, and horizons were changing’. 
(School V) 


‘... from the first trip we saw a big increase in confidence, engagement and also independence. Before, 
it had always been hard work to get them writing, particularly to get them to write at any length ona 
specific topic. But suddenly it seemed we had no reluctant writers. After the trip they were itching to 
write about it, and to write about it they needed to develop their skills.... | think that was the key to 
progress, them understanding that you need to learn rules about writing, because if you don’t you cant 
say what you want.’ (School W) 


Problems/limitations 


The first was time; it was felt by some that formally scheduling meetings between teachers may have 
extended the collaborative opportunities they enjoyed. Second, it was reported that the ‘absence of a 
real partnership working with partner schools’ was a missed opportunity. Third, ‘establishing a project 
with so many stakeholders was difficult’; the teachers needed persuasion; the students and parents 
needed sensitive engagement. (School A) 


It was felt that while the workshops were time-consuming and it was hard to arrange for the whole 
teaching team to be there, one thing they provided was quality time for discussion and reflection, and 
this time ‘would be better spent within school communities working on real plans, rather than rehearsing 
the issues in the workshops and then finding it difficult to make quality time available to take things 
forward back in school’. (School B) 


...a related problem she cited is the amount of time out of school it has required. She reports that a 
number of heads she has approached to participate in the project under her leadership next year have 
been interested, only to refuse ‘when it became clear how many hours out of school teachers would 
have to spend’. (School C) 


‘....the view was expressed that having teachers and leaders in the same workshop would make them 
more purposeful and improve communication. ‘if we were all in the same room at the same time it would 
make life easier. It is hard to communicate with each other once you get back into school. Maybe there 
are reasons to do with release, but it is not obvious why leaders and others cannot do these training 
sessions together’. (School D) 


‘...the academy gradually began to withdraw from the programme about halfway through the year. The 
reasons for this disengagement were summarised thus: Unsatisfactory experience within the local trio 
of schools The workshop programme’s (content) lack of relevance to the Primary sector generally The 
failure of the central training programme to reflect, or indeed show understanding of the specific 
challenges facing the academy The academy’s continuing staffing problems ‘ ( School F) 


‘Looking back, | wish we’d chosen Year Seven, because by Year Nine it’s too late. That’s what we’ve 
found anyway. The students we chose are making progress, but their starting point was too far behind 
by the start of the project to make enough difference’. ( School J) 


‘It would have been far better if our trio had been a trio of Primary schools ... because we were working 
on similar things, where it was difficult to find links between Primary and Secondary and the 
conversations that you were having when you were out on the training days..... you know, | just found 
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myself having to go and join the primaries so that we could put into our context what the Secondary’s 
had been saying in their conversations’. (School K) 


‘Nothing that's been done at the (HUB School) has been directly relevant to a Primary school. So the 
Primary school lead has really struggled to make the most of it. The materials, the PowerPoints; it's 
like they haven't really thought of Primary schools, specifically. It's like, “you're here too, so it'll be 
relevant’, but it's actually not.....’ (School L) 


‘....that when she had talked to colleagues from other schools who were in their second year of 
participation in the programme, she had been given the impression that the materials for the workshops 
and the content that was covered was the same. Helen commented that if this was the case, she would 
like to see materials developed for those schools who participated over a number of years, so that they 
were not just repeating the same things, but learning new things all the time’ (School M) 


‘...the cost is quite prohibitive to them (small Primary schools), both the cost to be part of the 
programme, but also the cover becomes more difficult if you’ve only got four teachers, you can‘ really 
get two or three of them out for a day’. (School O) 


‘Some of the training has been ‘disappointing’, even ‘patronising’, the school leader training sessions 
especially so. Much of material used did not seem very relevant to Primary schools, there was a need 
to re-design some activities and materials to reflect the participation of Primary schools, so ‘it seemed 
less like a Secondary initiative that primaries had merely been allowed into’ (School V) 


The training was viewed as interesting, but not really providing practical advice. In part this may have 
been because the majority of the local cluster schools were focusing on improvements in mathematics, 
and this school was in the only trio tackling literacy, so reports from other trios were not directly helpful. 
But beyond that there was a feeling that it was ‘spread too thin, nothing was covered in real depth’. And 
too much of the time was taken up filling in forms, ‘These were for the benefit of CtG, not for us’. There 
was also duplication, and insufficient opportunity to interact with other schools from the cluster, *... it 
underlines the benefits come from your own trio’. (School W) 


| Plan | Progress | Outcomes 


What do we intend to do What changes do we What progress in 

to produce an impact on expect to see that will attainment do we expect 
pupil progress and eventually produce an to see at the end of a 
attainment? impact on attainment? term? 


What progress in 


What evidence can we attainment do we expect 
gather for this? to see at the end of the 
year? 
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Sample 


In discussion with the project team, we have chosen three trios of schools, in different parts of the 
country, where we will develop accounts of what happens in each of the partner schools. During the 
year we will visit all these schools, at least twice, to collect information on activities arising from the 
project and to interview a range of staff, including some who have been less directly involved in the 
project strategies. 


Towards the end of the year we will prepare an account of practice for each trio, and these accounts 
will be shared with the particular schools. This will allow us to test the credibility of our analysis with 
stakeholders, whilst at the same time offering feedback that should promote reflection and professional 
development within the schools. 


The initial visit 


Our first visit to the schools will take place in January or February 2013. During this visit we would like 
to hold discussions with the project leader, plus at least one teacher and one member of the support 
staff who are closely involved in the project. In addition, we will be happy to meet other colleagues that 
schools wish to nominate, individually or in groups. 


The conversations will be guided by the questions set out above. The approach will be informal. 
Colleagues will be encouraged to focus on issues that seem to be of most relevance within the school 
and across their trio of schools. 


We would like to tape record the discussions. This is so that we can replay them later as we attempt 
to develop an account of what has happened so far. The names of individuals will not be included in 
these accounts, although inevitably it will be relatively easy to guess who said what at this stage. 


The evaluation team 


The team brings together a range of experience and expertise in relation to the themes of this particular 
project, as well as varied methodological strengths. The team members are as follows: 


Professor Mel West. He has carried out extensive research in relation to networking and collaboration, 
within and between schools, both for NCSL and for the DfE during the Leadership Incentive Grant 
initiative. He has a particular interest in aspects of management and leadership. 


Professor Mel Ainscow. He is internationally recognized for his research around the themes of inclusion 
and equity. He was also involved in the LIG implementation and subsequently as the Government’s 
Chief Advisor for the Greater Manchester Challenge, where extensive use was made of school-to- 
school collaboration. 
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Dr Michael Wigelsworth. He has been involved in evaluations measuring the impact of a series of 
national initiatives, including Primary SEAL, Secondary SEAL and Achievement for All, all which have 
involved the analysis and interpretation of impact from specific interventions on national data sets. 


Dr Trudi Martin. She has extensive experience in the implementation and evaluation of learning and 
development programmes, and is also a development coach. Her particular interests are the provision 
of education for children who have complex learning difficulties and knowledge sharing within and 
between schools. 
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Appendix 4: EEF cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention over 
three years. More information about the EEF’s approach to cost evaluation can be found here. Cost 
ratings are awarded as follows: 


Cost rating | Description | 


ELLEL Very low: less than £80 per pupil per year. 
LEELLE Low: up to about £200 per pupil per year. 


LEELLE Moderate: up to about £700 per pupil per year. 
LLLEL High: up to £1,200 per pupil per year. 
£EELE | Very high: over £1,200 per pupil per year. 
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Appendix 5: Padlock rating 


Primary 


attire) 


Ofdilat-Wielme-lilare| 


Well conducted experimental 
design with appropriate 
analysis 
4@ |Fairandclearquas- | | J | fo fo 
experimental design for 
comparison (e.g. RDD) with Adjustment 
appropriate analysis, or MDES< | 44.09% for 
experimental design with 0.3 Balance 
minor concerns about 
validity [-] 
3@ |Well-matchedcomparison [| | &4| J foo fo 
(using propensity score 
matching, or similar) or MDES < 
experimental design with 0.4 21-30% 
moderate concerns about 
validity ' 
tone Adjustment 
2a for threats 
MDES < to internal 
0.5 le validity 
a ee a Un] 
Comparison group with poor 
or no matching (E.g. MDES < 51-50% 
volunteer versus others) 0.6 
Oa 
No comparator ar > 350% 


e Initial padlock score: lowest of the three ratings for design, power and attrition = 2 padlocks 
e Reason for adjustment for balance (if made): n/a 

e Reason for adjustment for threats to validity (if made): n/a 

e Final padlock score: initial score adjusted for balance and internal validity = 2 padlocks 


“Attrition should be measured at the pupil level, even for cluster trials. 
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Secondary 


a ; iti F mle 
Criteria for rating Adjust 


Design 


5a Well conducted 
experimental design 
with appropriate 
analysis 


4a Fair and clear quasi- 
experimental design 
for comparison (e.g. 
RDD) with 


appropriate analysis, 11-20% 


or experimental Adjustment 
design with minor for Balance 
concerns about 

validity [-1] 


34 Well-matched 
comparison (using 
propensity score 
matching, or similar) MDES < 
or experimental 0.4 21-30% 
design with moderate ; 
concerns about Adjustment 
validity for threats to 
internal |. 
24 validity 
MDES < ” ; 
05 31-40% [-] 
18 Comparison group 
with poor or no 
matching (E.g. MDES < 51-50% 
volunteer versus 0.6 
others) 
Oe 
No comparator MDES > 
0.6 >50% 


e Initial padlock score: lowest of the three ratings for design, power and attrition = 2 @ 
e Reason for adjustment for balance (if made): difference of 0.058 on GCSE grades 
e Reason for adjustment for threats to validity (if made): n/a 

e Final padlock score: initial score adjusted for balance and internal validity = 1 @ 
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